The Kitchen in the Zoo - maxant tag

Building a Webmail Solution on top of Apache James Mail Server

Tue, 29 Sep 2009 13:49:21 GMT

Part of maxant's offering to small businesses is email hosting. As well as standard POP3/SMTP access, maxant offers webmail access. A quick search on the web shows that there are several open source webmail solutions available. The problem with all of them is that they communicate with the email server through the SMTP protocol. For example, if you wish to preview a list of emails, the web application needs to access the email server and ask for details of each email (while leaving them on the email server, so they can be downloaded at a later time via POP3).

Reading all the emails is inefficient and the larger the number of emails in your inbox, the longer it takes to just see a list of emails. The solution built by maxant is based on the Java Mail API from Sun. This API lets you access individual emails in your inbox using an ID. But Apache James Mail Server (James for short) doesn't maintain the index, if a new mail is put in the inbox, so if you have a list of all emails and decide to access one, and in the mean time you have received email, the chances are that you won't be able to read that email!

The next problem is how to deal with keeping a copy of sent emails for your "sent items" folder. If you just use the Java Mail API, the only solution for getting a mail into your email server so that it can be downloaded later via POP3 is to send that email explicitly to yourself, or to put yourself on CC. If you put yourself on CC, it gets confusing to some mail servers when replying to you, so that you get several copies of the email. The other problem is that that email will then end up in your inbox!

Another problem is message flags and folders. The Java Mail API offers the ability to add information to email messages such as the folder which they are in or details about when they were read, replied to or forwarded. But these flags are optional - email servers to not have to implement them, and indeed James does not.

There were several steps in solving these problems. The first was to set up a database repository for James so that emails were stored to a database rather than a file system, as is default. Having the email in the database allows one to access them using their primary key - a generated ID and the username. So if other email arrives in between reading a list of all email and drilling down to the detail of an individual email, you are guaranteed that you will always access the correct email.

In order to be able to store information about folders and message flags, and additional table was created with a foreign key to the message ID in the James table. Any information that needed to be stored additionally about emails could be stored there, such as the date and time of reading or replying.

To make reading a list of emails efficient, a selection of all emails in the users account was compared to their message flags. Where message flags were missing, full email details were read from the James table and used to create an entry in the other table. Since the other table was lighter weight (it only contained the information for displaying a list of emails), it was quicker to read and there was less data transfer between database and application, which is not only quicker, it causes less memory problems in the application.

When sending an email, a copy for the "sent items" folder was just created directly in the database as opposed to sending an copy of email to James for it to place it in the database. This got over the issues discussed above.

The only problem with all of these solutions was the requirement to use the James API within the web application in order to read/write directly from the database. James stores email messages as blobs in the database and to translate them into objects requires part of the James API. This is fine until a new version of James comes out in which case the libraries might not be compatible. That makes version migration an issue for this solution because of the additional rebuilding and testing required when upgrading the email server. However, James is a very solid and reliable piece of software and it is unlikely that such migrations will occur frequently!

So, to summarise, if you want a proper webmail client which interacts with the email server properly, first choose the right email server! In reality you will probably find that you need to integrate the two to a much higher degree than you intended, to get the effects you desire.

Copyright (c) 2009 Ant Kutschera

Social Bookmarks :

Blog Piracy...

Sat, 04 Oct 2008 11:45:00 GMT

The internet never ceases to amaze me... For it appears someone at ease with a cyrillic language is copying my blog word for word:

http://rational_software_architect.softblog.biz/category/work/page/2/

I guess I shall take it as a compliment. I also guess they are just an RSS subscriber and their blog gets updated automatically, so perhaps all of my future postings will have copyright notices in them :-(

So, this posting is Copyright 2008, Ant Kutschera!

UPDATE: Excellent - I see the site has been taken down! I wonder if its because of the email I sent to their provider, or because they just got bored? Probably the latter :-( If you notice that the site is ever up again, please post a comment below to inform me. Thanks!

Social Bookmarks :

The decline of dinos

Thu, 12 Jun 2008 20:11:00 GMT

No, not a report about the death of dinosaurs many millions of years ago, nor a report about how in fact many dinosaurs didn't die out, but rather evolved into birds...

Rather, this blog entry is about dinos which is a Facebook Application written by maxant. Basically, you get a virtual dinosaur which you need to feed. It can also fight or flirt with the dinosaurs of your friends, which directly affects its health. The idea is to keep your dino alive as long as possible.

There were several goals in developing it. First of all it was an experiment to see how the Facebook API worked. Second, it was to see how much faster a Facebook application would grow, compared to a normal website, because Facebook gives you "free website advertising" in terms of being able to invite friends to play your game/application. As such it is a type of viral marketing. The aim is to give it enough momentum that its growth will become exponential. At a minimum, growth should be linear shouldn't it? Well, right from the start, Google Analytics were used to track site traffic. Below are some graphs showing this traffic.

The general trend is slow but certain death, much like most original dinosaurs, all those years ago. The point of this blog is not to show that maxant is good at making crap games. Generally speaking Facebook application installations are falling fast, as the novelty wears off. The result is less new people playing the game and as long term players start to get bored, they just uninstall it. It's the same principal as pandas dying out because they don't mate and bear children frequently enough.

Now what's interesting about all that? Well there is a site called adonomics.com which tries to give valuations to Facebook applications (they also broker deals for application sales and do VC). Back in its hey day, dinos was valued at thousands of dollars. If someone had bought it (if indeed it had been for sale!) they would have hoped to earn revenue through advertising, to at least get a return on their investment, but hopefully to rake in much more! Originally and for only a short time, dinos had Google Ads on its page. After a short time, a much more lucrative advertiser came along namely cubics.com. During its lifetime, dinos has managed to earn roughly 50 US cents per thousand page views. It uses ethical advertising, so nothing where the user is forced or tricked into clicking on ads or where they are required to complete an ads questionnaire in order to help them in their game. And perhaps as a result, dinos hardly manages to pay its rent (that is, server costs). If development time, server costs and advertising revenue are summed, it bluntly never made a profit. Lucky for the individual who might have bought it based on the adonomics.com valuation, that it was never for sale!

If you look at bigger applications, who also use ethical advertising campaigns, their valuations are also well out. This is not because adonomics.com is hyping the applications values, it is because their calculations are probably based on the valuation of Facebook (an albeit high one).

You don't even have to go so far as looking at Facebook applications. Just take the market valuation of Facebook and divide the number of active users by it. It turns out (based roughly on figures I remember from last October) that an average Facebook user would need to generate hundreds if not thousands of dollars of revenue for people advertising on Facebook. Since I don't ever buy anything based on Facebook ads, and none of my friends do either, there must be some people spending thousands or tens of thousands of their hard earned cash on products being advertised on Facebook. Or maybe it is just seriously over valued. Or more likely, people/companies who advertise on Facebook are really really getting ripped off. Who knows?

But it does prove that making real money on the net is still really hard if you don't sell porn or host a gambling site.

Social Bookmarks :

Image Encryption II - Steganography

Thu, 19 Jul 2007 20:18:00 GMT

See the old post.

Apparently this is called steganography and is a well researched field! That just proves you how hard it is to invent anything new. Anyway, I have optimised the algorithm and build a website dedicated to it: http://stega.maxant.co.uk.

Social Bookmarks :

Image Encryption

Thu, 12 Jul 2007 22:01:00 GMT

Problem
The trouble with encrypting documents is that its quite obvious that they have been encrypted which makes people want to decrypt them.

So what about some kind of sexy algorithm that encodes a message within a picture? If the picture only gets slightly changed, then no one will ever realise it contains a secret message, since it looks to all intents and purposes like its a genuine picture.

Analysis
So, an evenings hacking later and I've come up with an algorithm that does it. The program takes an image file (gif, jpeg, etc) and outputs a bitmap (it cannot be output to a format that compresses with loss like JPEG, otherwise its not possible to decrypt the message since what gets saved is saved with bits lost).

Here is an example of a red image that has been modified to contain a message:

OK, if you look close (see below which is zoomed in) you can see the modified dots. But its just an example.

So here is a photo that has been modified to contain a different secret message:

Again, without zooming in, its hard to see the changes. Below is a zoomed in version. You can see the changes as speckled dots.

The algorithm has been tuned to try and match the changes to the pixels around the changed pixel. If the surrounding area is blue, the change will be blue, whereas if the area around the change is red, the result will be a red pixel. You should be able to see this quite easily. One thing I found was that its not so good if the image contains lots of white or black. The algorithm prefers red/green/blue images.

So if we use a standard photo editing program and add some noise (uniform, 20% in the following case), you really cannot see the changes that the algorithm has made. It simply looks like a badly taken picture.

It's even better if you use some kind of effect on the image:

During encryption I provide a pass phrase that is used to encrypt the message. The text is encrypted before being inserted into the image, so effectively it has been doubly encrypted. There is also a parameter that is used to specify how close together each changed pixel is, so that either all changes are to a single region, or all changes are spread out throughout the image.

In these examples only about a thousandth of the original image has been changed. With rough images, there would be no problem changing several percent of the image. But what is sure is that the resulting image (as in the encrypted hidden file) would be 20, 40 or even 100 times as large as the file or message being encrypted.

Practical Applications
The best application is simply put, file encryption, hiding the input within an image so its harder to find by prying eyes. The above example shows how text messages have been encrypted, but there is no reason this cannot be extended to files. How about a Web Service or Web Application that anyone could use to encrypt their secrets? Potentially coming up soon...

I'm quite sure this has been done before, but I just wondered how easy it would be... and at first glance, it seems not that hard.

If you're interested in its use, contact me.

And as enticement to crack this algorithm, how about 100 english pounds to the first person to email me the message that's encrypted in the speckled version of the picture of the clouds?

**** UPDATE ****

Apparently this is called steganography and is a well researched field! That just proves you how hard it is to invent anything new. Anyway, I have optimised the algorithm and build a website dedicated to it: http://stega.maxant.co.uk.

Social Bookmarks :