<< September 28, 2009 | Home | September 30, 2009 >>

Idempotency and Two Phase Commit

The requirement for services to be idempotent is often stated as being important for enterprise applications. But what does that mean, and why?

Idempotent means that a service can be called multiple times with the same data, and the result will always be the same. For example, if a service call results in a value being written to a database, the same service call made again would result in the same value being written to the database. As such, additative processes where values are incremented cannot be idempotent, for example an insert statement in a database is not idempotent, whereas an update statement usually is.

Imagine the case of purchasing a ticket from a web service offering airline tickets. The process probably includes getting an offer to see the price and tarif, reserving an instance of that ticket and finally when the shopping cart is full, confirming that ticket by booking it. Getting an offer would be an idempotent call, since we are just effectively reading data, not writing it. Reserving the ticket cannot be idempotent because each call should result in an individual seat being temporarily reserved - you don't want to reserve the same seat for two passengers. However, should such a reserved ticket not be booked, a background process would need to cancel the temporary reservation, so idempotency is effectively achieved. In the final call, to book a reservation (to guarantee the seat), the call should be idempotent - setting the status of the ticket to "booked" can be made over and over, it is not an additative process.

Why is it important? When it comes to booking, you need to make a payment to confirm the booking. That payment and the booking need to be done in a single transaction. There are two systems involved (booking system and payment system). That means either rolling back everything if either one of the booking or payment fails, or having the ability to redo one of the calls in case the other one fails and you want to try again later (immediately by clicking the "PAY" button, or perhaps in BPEL, at some time later, when the failed process is inspected). If you can rollback both calls, within an XA (two phase commit) transaction context, then you have no problem. But in this day and age, where foreign systems are called over non-transactional web services there is no XA context. So what you need to do, is have an idempotent booking service where you can call to make the booking again. You also need a background process which runs in order to explicitly cancel such bookings, if the payment really cannot be made. But in the time between the first failure, and the background process cancelling the ticket, you do need to be able to try and book again, which requires an idempotent service. If neither two phase commit nor idempotency are available, then in the case of a failure, your only choice is to cancel the ticket and make a new reservation and try and book it and pay for it.

So, the result is that when you integrate enterprise systems you really need to analyse the transaction concept in great detail, to ensure your customer gets the best experience in case of any system failures.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Building a Webmail Solution on top of Apache James Mail Server

Part of maxant's offering to small businesses is email hosting. As well as standard POP3/SMTP access, maxant offers webmail access. A quick search on the web shows that there are several open source webmail solutions available. The problem with all of them is that they communicate with the email server through the SMTP protocol. For example, if you wish to preview a list of emails, the web application needs to access the email server and ask for details of each email (while leaving them on the email server, so they can be downloaded at a later time via POP3).

Reading all the emails is inefficient and the larger the number of emails in your inbox, the longer it takes to just see a list of emails. The solution built by maxant is based on the Java Mail API from Sun. This API lets you access individual emails in your inbox using an ID. But Apache James Mail Server (James for short) doesn't maintain the index, if a new mail is put in the inbox, so if you have a list of all emails and decide to access one, and in the mean time you have received email, the chances are that you won't be able to read that email!

The next problem is how to deal with keeping a copy of sent emails for your "sent items" folder. If you just use the Java Mail API, the only solution for getting a mail into your email server so that it can be downloaded later via POP3 is to send that email explicitly to yourself, or to put yourself on CC. If you put yourself on CC, it gets confusing to some mail servers when replying to you, so that you get several copies of the email. The other problem is that that email will then end up in your inbox!

Another problem is message flags and folders. The Java Mail API offers the ability to add information to email messages such as the folder which they are in or details about when they were read, replied to or forwarded. But these flags are optional - email servers to not have to implement them, and indeed James does not.

There were several steps in solving these problems. The first was to set up a database repository for James so that emails were stored to a database rather than a file system, as is default. Having the email in the database allows one to access them using their primary key - a generated ID and the username. So if other email arrives in between reading a list of all email and drilling down to the detail of an individual email, you are guaranteed that you will always access the correct email.

In order to be able to store information about folders and message flags, and additional table was created with a foreign key to the message ID in the James table. Any information that needed to be stored additionally about emails could be stored there, such as the date and time of reading or replying.

To make reading a list of emails efficient, a selection of all emails in the users account was compared to their message flags. Where message flags were missing, full email details were read from the James table and used to create an entry in the other table. Since the other table was lighter weight (it only contained the information for displaying a list of emails), it was quicker to read and there was less data transfer between database and application, which is not only quicker, it causes less memory problems in the application.

When sending an email, a copy for the "sent items" folder was just created directly in the database as opposed to sending an copy of email to James for it to place it in the database. This got over the issues discussed above.

The only problem with all of these solutions was the requirement to use the James API within the web application in order to read/write directly from the database. James stores email messages as blobs in the database and to translate them into objects requires part of the James API. This is fine until a new version of James comes out in which case the libraries might not be compatible. That makes version migration an issue for this solution because of the additional rebuilding and testing required when upgrading the email server. However, James is a very solid and reliable piece of software and it is unlikely that such migrations will occur frequently!

So, to summarise, if you want a proper webmail client which interacts with the email server properly, first choose the right email server! In reality you will probably find that you need to integrate the two to a much higher degree than you intended, to get the effects you desire.


Copyright (c) 2009 Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!