<< August 2009 | Home | October 2009 >>

Idempotency and Two Phase Commit

The requirement for services to be idempotent is often stated as being important for enterprise applications. But what does that mean, and why?

Idempotent means that a service can be called multiple times with the same data, and the result will always be the same. For example, if a service call results in a value being written to a database, the same service call made again would result in the same value being written to the database. As such, additative processes where values are incremented cannot be idempotent, for example an insert statement in a database is not idempotent, whereas an update statement usually is.

Imagine the case of purchasing a ticket from a web service offering airline tickets. The process probably includes getting an offer to see the price and tarif, reserving an instance of that ticket and finally when the shopping cart is full, confirming that ticket by booking it. Getting an offer would be an idempotent call, since we are just effectively reading data, not writing it. Reserving the ticket cannot be idempotent because each call should result in an individual seat being temporarily reserved - you don't want to reserve the same seat for two passengers. However, should such a reserved ticket not be booked, a background process would need to cancel the temporary reservation, so idempotency is effectively achieved. In the final call, to book a reservation (to guarantee the seat), the call should be idempotent - setting the status of the ticket to "booked" can be made over and over, it is not an additative process.

Why is it important? When it comes to booking, you need to make a payment to confirm the booking. That payment and the booking need to be done in a single transaction. There are two systems involved (booking system and payment system). That means either rolling back everything if either one of the booking or payment fails, or having the ability to redo one of the calls in case the other one fails and you want to try again later (immediately by clicking the "PAY" button, or perhaps in BPEL, at some time later, when the failed process is inspected). If you can rollback both calls, within an XA (two phase commit) transaction context, then you have no problem. But in this day and age, where foreign systems are called over non-transactional web services there is no XA context. So what you need to do, is have an idempotent booking service where you can call to make the booking again. You also need a background process which runs in order to explicitly cancel such bookings, if the payment really cannot be made. But in the time between the first failure, and the background process cancelling the ticket, you do need to be able to try and book again, which requires an idempotent service. If neither two phase commit nor idempotency are available, then in the case of a failure, your only choice is to cancel the ticket and make a new reservation and try and book it and pay for it.

So, the result is that when you integrate enterprise systems you really need to analyse the transaction concept in great detail, to ensure your customer gets the best experience in case of any system failures.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Building a Webmail Solution on top of Apache James Mail Server

Part of maxant's offering to small businesses is email hosting. As well as standard POP3/SMTP access, maxant offers webmail access. A quick search on the web shows that there are several open source webmail solutions available. The problem with all of them is that they communicate with the email server through the SMTP protocol. For example, if you wish to preview a list of emails, the web application needs to access the email server and ask for details of each email (while leaving them on the email server, so they can be downloaded at a later time via POP3).

Reading all the emails is inefficient and the larger the number of emails in your inbox, the longer it takes to just see a list of emails. The solution built by maxant is based on the Java Mail API from Sun. This API lets you access individual emails in your inbox using an ID. But Apache James Mail Server (James for short) doesn't maintain the index, if a new mail is put in the inbox, so if you have a list of all emails and decide to access one, and in the mean time you have received email, the chances are that you won't be able to read that email!

The next problem is how to deal with keeping a copy of sent emails for your "sent items" folder. If you just use the Java Mail API, the only solution for getting a mail into your email server so that it can be downloaded later via POP3 is to send that email explicitly to yourself, or to put yourself on CC. If you put yourself on CC, it gets confusing to some mail servers when replying to you, so that you get several copies of the email. The other problem is that that email will then end up in your inbox!

Another problem is message flags and folders. The Java Mail API offers the ability to add information to email messages such as the folder which they are in or details about when they were read, replied to or forwarded. But these flags are optional - email servers to not have to implement them, and indeed James does not.

There were several steps in solving these problems. The first was to set up a database repository for James so that emails were stored to a database rather than a file system, as is default. Having the email in the database allows one to access them using their primary key - a generated ID and the username. So if other email arrives in between reading a list of all email and drilling down to the detail of an individual email, you are guaranteed that you will always access the correct email.

In order to be able to store information about folders and message flags, and additional table was created with a foreign key to the message ID in the James table. Any information that needed to be stored additionally about emails could be stored there, such as the date and time of reading or replying.

To make reading a list of emails efficient, a selection of all emails in the users account was compared to their message flags. Where message flags were missing, full email details were read from the James table and used to create an entry in the other table. Since the other table was lighter weight (it only contained the information for displaying a list of emails), it was quicker to read and there was less data transfer between database and application, which is not only quicker, it causes less memory problems in the application.

When sending an email, a copy for the "sent items" folder was just created directly in the database as opposed to sending an copy of email to James for it to place it in the database. This got over the issues discussed above.

The only problem with all of these solutions was the requirement to use the James API within the web application in order to read/write directly from the database. James stores email messages as blobs in the database and to translate them into objects requires part of the James API. This is fine until a new version of James comes out in which case the libraries might not be compatible. That makes version migration an issue for this solution because of the additional rebuilding and testing required when upgrading the email server. However, James is a very solid and reliable piece of software and it is unlikely that such migrations will occur frequently!

So, to summarise, if you want a proper webmail client which interacts with the email server properly, first choose the right email server! In reality you will probably find that you need to integrate the two to a much higher degree than you intended, to get the effects you desire.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Creativity - Yet another driver towards agile processes

Software developers are often creative people, having entered into software development precisely because of its need for creativity during implementation. While solving a logical problem might sound unrelated to creativity, the ability to solve that problem in many different ways and choosing the right way, is what requires the creativity. This is what turns a mundane data entry job into an exciting and fun job.

Not only are there many different ways to implement algorithms or user interfaces, so that they provide the same inputs and outputs as required, there are many different ways depending upon the architectural or design view points taken. For example, should the implementation use as much reusable code as possible, or should the code be as simple, readable and hence maintainable as possible? Or is breaking architectural policy allowed, in order to ensure maintainability, productivity or other aspects, etc...

So if programmers, designers and architects are so creative, is it a problem? Well it can be, when requirements specifications are rigid. Developers need a certain amount of leeway in order to feel that they are being creative whilst doing their job. If a specification goes into exact detail about how to implement something (which a requirements specification shouldn't actually need to do, unless the requirement is defining the inputs or outputs of the system), then a developer can feel there is no need for creativity, and the fun part of their job disappears quickly. Once that happens, developer motiviation becomes reduced and project success starts to suffer. Indeed a measure of software success is the level of developer happiness at the end, because you need these people to be motivated for the next phase or project.

The alternative to rigid requirements is a more agile approach. If the customer* can work with the development team in an agile manner, defining the outcome of the software together and allowing the development team to provide input on say usability aspects, or technological options, then everyone is a winner. The developers feel creative and the customer gets something which he can definitely use and definitely wants, because the agile process is an iterative one with short cycles between feedback and response (delivery of an updated version of the system). Often when writing rigid specifications, they suffer from "specification rot" (exactly the same thing as code rot, which is when code quality reduces over successive iterations of changes because new changes fail to take old ones into consideration). Basically said, the specifications become incomplete, containing conflicting requirements and illogical portions. Close and agile work between customer and development team allows rigid specifications to become living documents which can be adapted and perfected.

An advantage to the customer is also that they may become aware of other possible solutions to their problem. For example there may be a cooler way to make the user interface interact with the users. "Cool" can be a very important metric when it comes to web site design. Don't believe me? Take a look at iPhone sales related to the "cool" factor. Developers may also offer better or more efficient ways to solving the customers problems.

Of course, there are two sides to any coin, so agile processes must have disadvantages. Here are a few:

    - Scope Creep - the customer realises quickly with each delivery that they need something other than what they have specified. But I have experienced plenty of projects where the customer doesn't realise that, until six months or a year down the line and by then all of their budget has been spent and it's too late to fix the problem. At least in an agile process, the customer specifies through priorities, what is most important to them within their fixed budget, and that priority list is allowed to change as the customer discovers the system.

    - Rigid release cycles - large corporations whose software is all interlinked may use say two fixed releases a year in order to synchronise releases and versions of software in order to reduce the complexity of running multiple versions of the same software at the same time. While such strategies do indeed make sense, there is no reason why individual projects may not be run lean and agile. The customer can certainly be given access to their software frequently as it is being built. The excuse that the software is not ready because interfaces to other systems in the big release cycle are not ready does not fly, because otherwise the software could not be built until those systems are completed (using stubs, mocks or similar are the solution here).
    - Reluctance to change - many customers who have aquired software in the past only know how to do it using rigid and old fashion methods. It is quite human to dis-like change and go with what you know. But I encourage you to try going agile on your next project, even if it is only to prove me wrong, that agile projects are more successful than rigid ones.
    - Inexperienced customers - to be a useful and good customer on an agile project requires that you are open to change and suggestion. Many customers create a picture in their mind of what they think they want and define that in a rigid way. They are afraid that if their rigid requirements are not implemented that the software will be wrong in the end. Well here's the news: it will be anyway! There are very few individuals out there who a) know what they really need without going through an iterative cycle of trying it and then improving it, and b)  who even if they do indeed know what they really need, can communicate it so that it gets implemented that way. So instead of going through the disappointment of only playing with it when its finished, try playing with it as it comes to life, just like a new home owner inspects their house as it is being built, or an engineer tests their mechanics as they are built. In aeronautical engineering, the "customer" is generally considered to be the company designing the new plane, rather than airlines who purchase the planes. No aeronautical engineering outfit has ever built the entire plane before testing that the waste disposal systems worked as required! Indeed such systems are put through acceptance tests before they are even integrated into the airplane.

Out of these, the two important ones are inexperienced customers and scope creep / change. Prioritising EVERY TASK on the project is the key to keeping the project on track. Experienced customers know that arguing over the layout of the UI down to the last pixel while important, may be costly. They can decide if it is more important to the project (note: not them as individuals) where the priorities lie. Experienced agile developers know this too and together the customer and developers can ensure a successful project.

So, as the world moves on, there are still many non-agile and rigid projects happening throughout the world. If you are in charge of software requisition and have some budget to start a new project, why not try making the project an agile one, if only because the creativity involved makes it a more fun project!

Let's quickly define a "customer". In the sense of software development and aquisition, this group of people are the stake holders who are paying for the work to be done. They are not the users although should be part of that group in order to be able to specify what is really needed. In a world where a software product is to be built which will be sold to users, for example like what Microsoft's core business does, the customer here is not the end customer, but rather the group of people who decide what should be sold in the first place. The customer group typically includes business types who are in charge of the money and strategy, as well as people representing the user community.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

NEWS FLASH: Agile does not mean there is no plan

Why aren't all projects run using agile methods? Working in an enterprise where we have maybe 50 applications which all go live at the same time twice a year (as well as another 450 which are also in some stage of the software lifecycle), I was told that it would not be possible to plan such a release without tight control over the planning, and frankly agile methods do not cater for such planning...

Erm... I see it a little different. Agile to me means that there is a plan and you know the maximum amount of money that you can to spend. To me it also means there is a fixed deadline. As quality is also fixed (come on, it has to be bug free!), it is scope which changes. Or admittedly to some degree also quality, as you decide that some bugs are tolerable and you can live with them.

What is different about agile projects, is that the project plan and scope (functionality) are alive. They are dynamic. They change from day to day as everyone on the project learns and discovers. Just like your day to day plans change as events unfold.

It should be illegal for a project manager to state that a project plan is fixed. It shouldn't be allowed that the project manager, upon discovering that a plan needs to change, states that it will not be changed because it has already been communicated to the customer. This view is certainly un-agile. But it is also simply wrong! It is like saying "I have stated my intentions to walk to the moon by Christmas, and I shall do so, regardless of whether it is possible". In my world, it is very acceptable to admit your plan is wrong, and to re-state your intentions, especially if the deadline doesn't actually change ("I shall now fly there instead!").

Transparency to the customer, that you have discovered something that you did not put in your original plan, shows the customer that he can have faith in you, for being able to be honest about it and to re-plan, being able to handle such events.

The customer must also be able to tolerate such unplanned events, and doing so shows that the customer is mature enough to understand that this happens in software development. It also happens through out life, in all kinds or projects. When was the last time that a building was built on time? Planning is done wrong in all walks of life, and it is perfectly natural to re-plan. We should not be scared of admitting that our plans are wrong and we should certainly not be working in environments where one feels the need to hide such facts.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!