My opinion on technical debt

Definition of debt.

Speaking for a debt, we have to know that there are two participating parties: The debtor and the creditor. The debtor has an idea, and needs some assets to realize it. These assets are not immediately available to the debtor, but they are estimated to be available to him in a computed amount of time. Here is the point where the creditor enters the game. The creditor has immediately available these assets that are needed by the debtor, and he wants to sell them to him. Of course, the transaction cannot be completed once-off, so the two parties agree on an arrangement. The motivation of the creditor to start with this arrangement is the interest that will be derived. This is an investment for the creditor, and a bet for the debtor.

The debtor is now facing the challenge that he should succeed with his idea, so, he can return the value of the assets plus the interest, and of course, he should be able to make a profit for his own effort.

Metaphorically, everyone has an idea in any time of his life, and he always has to make decisions while realizing it. Right decisions are proven to be those that result to a profitable outcome. Profitable outcome of course means that the returning value of the assets and their interests are safe, being a wide variety of possible things, like ethics. Good ethics mean there is trust of the environment to the debtor, and the debtor can count on it in the future.

Software development economy.

Software development is usually driven by contracts between clients and software houses (software service providers). There is a system of managers, marketing teams and accountants etc. except from the technical team, participating to a project. And finally there are governments with tax returns. Both clients and service providers are obliged to return taxes while working for the success of their projects.

So, inevitably, in a system that every process is translated into money (salaries, contracts values, profits, taxes), decisions are driven by money also.

A client asks for a specific (the best he can do) solution to some problems, and he is making an investment, in order to make some profit. The software service provider offers the solution for a profitable price. When the contract is signed, the deadlines are the most important effects of the project for the client.

Parallelism of technical debt in software development.

Technical debt is of course a matter of work ethics. But it is not only that.

When a new contract is signed, a war starts at the premises of the software service provider. The contract may already have been evaluated, but in many cases only pre-sales department is familiar with it. Technical teams evaluate the project in order to estimate if the deadlines are feasible. I remind you that the contract is already signed, and of course it contains some deadlines, but technical teams hardly know that. Managers brainstorming team will evaluate technical estimations and they are responsible for the final agreement of the time-plan. The best manager will have the best idea: the new software solution may be a compilation of older legacy projects that have already been tested on production environments, so, there is a considerable reduction of implementation and testing efforts. Integration implementation and testing is usually a hidden cost to everyone, except from the developers. Managers’ favorite word in software houses are modules. And because they call them like this, they thing that it’s always very easy to be configurable and reusable – loosely coupled with a specific product.

Decisions are the drivers of our path through any attempt, even if they are technical decisions concerning the development of software solutions. Unfortunately, decisions in such cases are taken by the service provider’s managers because they are considered to be higher in hierarchy than senior developers or even architects. The reason is that between the roles of a manager are also some financial responsibilities.

Technical debt may be derived from technical decisions and/or financial decisions combined. Technical debt means that a project is technically inefficient because it costs too much to maintain and to evolve. There are technical and financial terms together in the same sentence (isn’t it great?).

And here comes the quality of the software. Definitely, a software solution or a software provider with a technical debt is not considered to be able to offer high quality of products/services. Wrong or biased decisions in financial and/or in technical aspects lead to low quality of products/services. And why is it called a technical debt? Because someone is called to deal with the defective product/service, working overtime, sacrificing weekends and over-negotiating to convince clients. The product/service by itself is hardly making money for the investor and his interest. The service provider is pushed to work extra for his profits. This extra work may be offered by the technical team, or by the managers’ team: technical team can work overtime for months in order to improve the quality, or the managers can be super-convincing negotiators and they can guarantee sustainability of a low quality product/service.

Conclusion.

Finally, I have to mention that hard problems need good and experienced solvers.

A system that has to be built from scratch with well written, frozen functional specifications, defining solid and clear functionality is the perfect task for junior developers. Of course we need some senior developers and an architect at least for any serious software solution delivery, but the majority of the team members should be, and can be, junior developers.

Senior developers are better and more appropriate for solving difficult problems. When the software is a mess, and the deadlines are strict, a developer should avoid a lot of pitfalls and face some challenges. Experienced developers, with strong problem solving abilities are more than suitable then. That’s why senior developers get paid better. In other words, technical debt is a decisions problem that immerse sooner or later in every software service provider. And the key to resolve this problem is to have capable and experienced members in the working team.

In a world with perfectly defined and frozen functional specifications, with frameworks and platforms that are fully compatible with each other, senior developers would be a luxury, and technical debt would be an artificial literary term.

Infrastructure for e-commerce web applications

Assets management system

In my previous job, I met a very good professional software developer who was considered to be the most experienced on the field of assets management. In the past he participated in a team developing an award winning assets management system. It was then when I realized the importance of this kind of systems. I realized that the image uploading web sites are actually assets management systems. Assets in their case are pictures.

Thinking deeper and cleaner, somebody could realize that a blogging web site is also an assets management system. A blog post is considered to be the asset in this case. It’s called a content management system because the asset is a more complex structure than monolithic binary data of images, and because there are features of presentation of assets (blog posts), filtering according to their properties and designing pages that include lists of assets.

Video management web sites are also asset management systems. In this case the assets are video files which, are assigned searchable properties, offer online editing, offer recommendation engines that lead the visitors to other videos of the same provider etc. Services vary, of course, by provider.

Thinking more widely, even an online store is an assets management system. Every product that is available for sale is an asset, and the web site is offering several features of searching and recommending assets (products) with the target of selling as much as possible.

Databases for assets persistence

Viewing everything that is served on the net as an asset, then databases can be seen as assets management systems from an application’s point of view.

Relational databases hold data tables that contain entries that describe entities. In most cases a set of joined tables describe a more complete entity. This entity may be considered an asset for an application. Object relational mapping (ORM) frameworks help developers handle multiple joins transparently, and consider a complex entity as a high level object that can be served by the application as an asset.

Architecturally simpler is the concept of the NoSQL movement. Object databases can store both structured and unstructured data, together with their properties and with the ability to change the structure of the object at any time of the application lifetime.

Provided services/functionality

1. Store an asset 
2. Retrieve an asset

User management system.

Users are entities that participate more actively in an application. They can be considered as assets to applications like social media and to applications that maintain a profile for each user. The most commonly used purposes of users are the access rights management that may be related to licensing, persistence of preferences and personalization of content.

Provided services/functionality

1. User registration 
2. Check login

Campaigns e-mail management system

Newsletters sending is a very popular way of advertisement. In a newsletter, a possible customer can be updated about the latest products and offers of a known company, without even visiting their site. This is the magic power of e-mail. Checking your email, feeds from your subscriptions arrive, without even visiting any other web-site. Of course the e-mail is linked with particular pages of the site, and the clicks on the newsletter are tracked. Customers relationship management system will be very happy to know that a customer is interested in a particular offer/product.

Another very important feature of a well designed and responsible system is the notifications engine and the most popular way of notifications is e-mail. The customer should be notified about his orders, and additionally he could set notifications on custom events provided on the web-site (e.g. notification when a product is available again, notification about changes of a product's price, notification when a new product is entered.)

Maybe the above functionality is more appropriately included in the features of the customers relationship management system, or the user management system. E.g. whether a user allows the application to send him e-mails maybe better included in the user management system. When a customer is interested in a particular product and he needs notifications on that product, then the customer relationship management system is more appropriate to include this information. The e-mail management system should probably remain agnostic of the business logic, and execute simple commands like sending a given e-mail to a given list of recipients.

Warning, I have seen some newsletters including the list of recipients in the 'To' field of the e-mail. Please include the list of recipients in the 'Bc' field of the e-mail, or send the e-mail in a loop including a single recipient at a time. Sending the newsletter in a loop using single 'To' recipients may cause a problem on your mail server with too many connections.

Provided services/functionality

1. Subscribe to the newsletter 
2. Unsubscribe from the newsletters
3. Send this content to this list of recipients

Payments system.

Of course the target of an e-Commerce application is the completion of an order and the payment from the customer. Payments are procedures that are offered by financial banking credit systems, and an e-Commerce application needs an interconnection with such a system. Payments are usually time consuming and asynchronous processes and these systems usually offer monitoring features of the progress.

Provided services/functionality

1. Submit payment with credit card for specific amount and user
2. Submit payment with bank wire-transfer for specific amount and user
3. Submit payment with online payment provider (paypal)
4. Get status of payment for an order

Shopping cart system.

Shopping cart is actually a list of selected products by a user that can be updated as he uses an e-commerce application. The cart can be only one for every user, and sometimes some applications may offer multiple carts as main or draft or temporary.

Provided services/functionality

1. Set assets management system 
2. Add product to the basket 
3. Delete product from the basket 
4. Change amount number of items for product 
5. Get the data of the basket 
6. Clear basket

Customers relationship management.

This is the system that holds the orders of the customers. It may also hold pre-orders, and calculate possible discounts. This is generally a history of transactions.

Provided services/functionality 

1. Create order 
2. Get list of orders with criteria 
3. Get discount for this shopping basket

Advertisements serving system.

Advertisements may be another additional way of profit for an e-commerce application. Suppliers may need an extra boost of promotion of their brand name or a specific product. Users may be interested for personalized advertisement messages, according to their previous purchases, the contents of their shopping cart (or carts), and of course according to their demographic data.

Provided services/functionality

1. Store an advertisement and its properties 
2. Retrieve a specific advertisement 
3. Retrieve a specific advertisement with parameterisation 
4. Retrieve an advertisement according to keywords and other criteria

Introduction to CV for application for a Master's degree program

As a professional software engineer with over 8 years of experience, I always demand cutting edge tools and the best software solutions design and implementation from myself.

I am a continuous learner and I love experimenting with everything new in software technology. I am also interested in human computer interaction, and in building hardware embedded systems.

I have designed and implemented multi-threaded communication protocol servers and graphic representation systems used in real world for medical and statistical purposes.

I was also involved in integration business projects, creating database management software systems with intuitive user interfaces and complex business logic rules.

I recently participated in a team developing an ambitious e-commerce site using cutting edge technologies in the domains of relational databases, document databases, message queues, data mining techniques and web services.

My dream project is an automated database persistence system that would adapt to any number of joined relational and document databases through a process of a few easy steps for an administrator. Later, the users would be able to define, design and change data views, create custom filters for data reports, and change data records. Also, users would be able to define data flows and business rules using the web application. Furthermore, the application would be able to notify the users when certain events are triggered, with reports containing the appropriate amount of information.

I am interested in extending my skills in the domains of robotics and artificial intelligence. I would like to develop a web management system with advanced monitoring features for robotic devices. I want to learn programming micro-controllers for real time throughput on robotic devices. Robotic devices may include advanced sensors like video cameras with pattern recognition technology and 3 dimensional moving parts. The final application would be an industrial automated expert system like a production line with minimum human intervention.

Data analysis is another challenging quality that I am interested in working at. Managing large data stores seeking for metrics and statistical analysis results may be interesting for business intelligence concerning decision support systems. Data is an asset for today's business, and information derived from them are valuable products. Data mining and combination of results can be very useful for strategic management and financial decisions.

Contemporary applications of any purpose are always supported with the appropriate mobile application. Working on the move is a growing trend, and mobile devices are now even replacing personal computers for some uses. Android, IOS, Blackberry and windows are very popular mobile platforms that can support web browsing, graphics display and interaction, and save time with their special minimal design.

Finally, I would like to state that I consider it a great challenge every opportunity to transform an old fashioned organization into a contemporary paperless working place, using the technologies of Internet communications, cloud computing, portable devices, expert systems and robotic automations. I greatly appreciate the environmental benefits of these transformations and the advances of quality of life and work for people working in these organizations. I really believe that information technology is a leading tool towards making right and better decisions in general.

USB flash disk history

Back in year 2000 the best way (and the only way) to transfer large amount of data was the recordable cd-roms. The cd burner devices were very popular and fashionable and everything was burnt on a cool compact disc.
I managed to buy a yamaha cd burner and I was very happy because I could keep backups of my work. But the problem was that I couldn't pay for dial up calls or an ISDN line, and I needed a way get some documents home from the office. Burning a cd for this purpose was an overkill because:
1. only one pc, in the company, had a cd burner,
2. I wanted to trasnfer documents of 5-10 megabytes size, and 650MB of a cd was a waste of resources.

So, then, in 2002, I found the solution called usb flash disc. My first usb flash disk was a sexy designed curved device, with 32 megabytes capacity.

I was proud for my portable device, as it costed about 40 euros. Then I was able to transfer my documents with an efficient way and without the need of having access to the internet from home. With this disc I managed to maintain a collection of programming articles and resources, that I was able to read from home. I had great time with my ripped articles, and the collection was always growing.

After that great step of progress, the need of buying a new hard disk drive appeared. My old hard disk of 8.2 Gigabytes was very small for my new expectations. Then I baught a 40 GB hard disk about 118 euros cost, and it was then when I downloaded a new bios for my old pc in order to support hard disks over 32 GB.

The new hard disk seemed to be a black hole for dropped in data. Data seemed to get lost in the vast spaces of that hard disk. So, I needed a transportation unit that could give a bigger bite to the hard disk capacity. Then I baught my second usb flash disk, that I used to call it usb FLASK disc, which had a capacity of 2 GB and I don't remember its price back then in 2004 (or 2006).

At last, I could transfer videos and pictures home. I started a new collection of data, called "Cool pics" and "Cool videos" of course... And the "new" hard disk drive now seemed to be small. Then, I started to backup data in cds and purchased dvd burner for more efficient backups.

Concurrency issues in real life

"Non-determinism comes from parallel processing and mutability" (Martin Odersky)... true for information systems and real life also.
This was the tweet I posted on midnight of 27th July 2011 to my twitter followers, while watching this video: O'Reilly OSCON Java 2011, Martin Odersky, "Working Hard to Keep It Simple".
I immediately thought that this phrase is a compressed dose of a great portion of wisdom. The art of information systems architecture and real life seem to have many things in common.

Real life could be multithreaded:

Thread 1: Health
I consider health to be the most important thing in our lives. And health depends of course of the conditions in the environment where we live. We live in a city and we prefer to spend some weekends and some holidays at the country side. We get sweat in the summer and we get cold in the winter interchangeably. The quality of the air we breathe and the water we drink is changing. And of course we need more regular physical exercising.

Thread 2: Family
Being a man, you know that your wife's character is more floating than you would expect. Living with another person always takes some tries to keep the moods in sync. Problems begin when one person is equiped with happiness while the other is seized by sadness.

Thread 3: Job
As a dangerously enormous part of our timeline, job significantly qualifies our lives. We may feel productive or we may feel stressed from a deadline. We may be satisfied of the subject, the conditions and the colleagues, or we may need a change of workplace, or even a change in our career.

Thread 4: Friends
Friends bring a great balance to our lives. Like family persons, friends are also persons who need to get and who can give sadness and happiness interchangeably.

Thread 5: Money
Since everything in life ended up to be needed to be measured somehow, we got money, and we got selling, buying, lenting and renting transactions. Sometimes, for some months, your salary is not enough to pay the bills. Some other times, unfortunately fewer, you may be surprised of the balance of your account, because you managed to spend less in that month.

And there are more than 5 "threads" in real life, all manageing:
a. mutable qualitative data as emotional states and
b. mutable quantitative data as money.
Unfortunately our brain is one "physical processor" with several "logical cores" but all of them have to use the same "shared memory"... So, all people have to deal with mutable data while processing threads in parallel.
Apparently, real life is naturally non-deterministic because it has to deal with parallel processing of mutable data. But, a software engineer should design information systems in a completely different way. An information system should be deterministic, so, it has to build a shield against parallel processing and mutable data.

Concluding, a deterministic information system allows the software engineer to:
1. take more time to take care of his health,
2. spend more time with his wife,
3. prepare better deliverables in time,
4. drink beers with his friends,
5. produce more in less time.

A year full of passion

I just turned an exciting year on my current job at www.xe.gr. I feel that much happened during this year, and now it's time for an account of things done, and scheduling tasks to be done in the future.

During this year:
1. I contributed to an e-commerce application and its shopping cart.
2. I got introduced to the Restless services, and implemented a restless API.
3. I found out that there are JSP alternatives, that are better for front-end developers.
4. I discovered the power of dynamic languages. Perl, Python and Scala are now in my immediate interests.
5. I realised that Tomcat servlet container can do the job as an "application server". JBoss, Websphere and Weblogic are great application servers for those who don't care about the overhead (or configuration) of preinstalled frameworks.
6. I dropped my absolute belief to relational databases as Oracle and MySQL, as I discovered NoSQL databases as MongoDB and Cassandra.
7. Now I am aware of over a hundred open source projects that can be usefull and I must use during my career.
8. I studied many lines of bad written legacy code, and I supported it.
9. I was introduced to the agile development process, and especially to Extreme Programming.
10. I used 2 major bug tracking systems professionally: Bugzilla and JIRA.
11. I created an account to a social network for professionals: linkedin.com
12. I created my account on twitter and twitted about 50 times.
13. I dropped an eye out of the box and I saw Javascript can be a server side programming language.
14. I decided to regulary attend the meetings of 2 user groups: jhug (Java Hellenic User Group) and greecejs (greek javascript user group).
15. I first used Maven as the build tool of my projects.
16. I extensivelly used subversion for versioning.
17. I created my account at github, started 2 new open source projects and started learning git for versioning.
18. I recently acquired an android smartphone, so, I started using mobile internet and I read more using the kindle application.
19. Now I can see much more potential in the smart mobile devices, and as a result I care more about mobile software development.
20. I purchased about 15 ebooks and 5 printed books about software development.

Phew! Pretty awsome and fascinating year!
And there is more...
I met great people with great personalities, who communicated their invaluable experiences and their interestingly different point of view.
I made some friends, and I spent great time lunching and drinking beers.

At the personal level, I admit that the current and the next year will be the greatest years of my life because in a few months I will take the greatest responsibility of my life: being a father.

Importance of Testing

To all hard working programmers around the world, I should verify this principle:
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.
It wont work unless it passes extensive testing.

No, this is not a blog post full of unnecessary repetitive nonsense. It's a priceless truth that should be burnt in your mind as you decided to be a coder for a living.
Don't ever, EVER forget this.