<< August 2008 | Home | October 2008 >>

ThreadLocal and Thread Pools

ThreadLocal is a mechanism that can be used to put data onto a thread so that it can be accessed from any code using that thread. One use is to make contextual information relevant to the current thread available throughout the stack trace, for example a transaction context, or security context.

If you use it yourself, you need to watch out if the environment in which it runs gets its
threads from a thread pool (e.g. if your application runs in a managed container such as in an app server).

Typically a thread pool gives no guarantee that the ThreadLocal (or indeed other internal state) will be cleared next time the thread is taken from the thread pool. This means you could have an issue  - consider this case:

Two web applications deployed in the same EAR (or even seperate EARs depending upon your class loader configuration and deployment) - both set a thread local which is a map (key / value pairs). One application, in a given case, processing a web request doesn't set a value in the map and calls a service which reads that unset value using a key (the same key is used within both applications). Instead of being unset (null / empty), that unset value could be a value from the other application, if the thread is not new and your code does not clean up the thread local before returning the thread to the pool.

I did some tests and these were the results:

  • Tomcat 5.5 - two web applications deployed independently: No problem because each web app uses its own class loader, so the thread local is unique to each application.
  • JBoss 4.2 - two EARs deployed independently: PROBLEM - it appears that both EARs are using the same class loader for the thread local, so it is shared between the applications.
  • Websphere 6.1 - two ears deployed independently: No problem, UNLESS you use the PARENT_FIRST class loader setting AND the thread local is in a library on a shared classpath (i.e. the thread local is loaded by a shared class loader like the Websphere Extensions Classloader rather than the CompoundClassLoader used for applications. The same problem could occur if the WebSphere is configured to use the same class loader for all web applications.

Here are some rules which should help you to use ThreadLocal safely in an environment which uses thread pools:

  1. Only use ThreadLocals where you are certain the context is within a single call to the server (e.g. it cannot be used to store state between calls).
  2. If you use a ThreadLocal, ensure you set it somewhere in the call chain - you cannot assume it is unset, just because you did not set it in the current call (this is similar to the Java Compiler forcing you to initialise variables)
  3. Clear up your ThreadLocal before it is put back in the thread pool, e.g. at the end of a doPost() method in a servlet.
  4. If your ThreadLocal is a map of properties, then use keys which are unique to the application (e.g. they contain the application name in them).

Another interesting point is that you aren't really meant to be doing things with threads within an application server anyway and as one of the postings below points out, Sun never really intended for ThreadLocals to be used in such environments.

A different approach to using ThreadLocal is to have a class in your application which contains a static Map where the key is a reference to the thread and the value is whatever you would otherwise have put in a ThreadLocal. So in this approach, you do not use ThreadLocal, you use the static Map. To get the data which is relevant to the current thread, you just do something like (assuming the data in the static map is another map):

    Map<String, String> localData = MyThreadLocalData.
           getStaticMap().get(Thread.currentThread());
    String data = localData.get(keyContainingApplicationName);

The problems with this approach are:

  1. You have references to threads which may eventually be released from the thread pool, but which cannot be garbage collected until you remove the reference to them in your static map. So ENSURE you clean up the static map when you are done with thread (or write a routine to automatically clear up - which gets a reference to all threads in the system from Thread.getAllStackTraces().entrySet() - any references to threads you have which are not in that set can be removed). If the static map is a WeakHashMap, then this problem shouldn't occur (thanks to Heinz Kabutz for the tip).
  2. If the MyThreadLocalData is loaded by a class loader which is shared between applications, you have the danger that one application can update another... (hence the reason to use application specific keys in the map of data contained in the static map).

Attached here, is a ZIP containing two EARs which I used to test all of this. The bored yet interested reader may download, study and play with them as an exercise... Have fun!


PS - Here are some other postings which back up these statements:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6254531

http://osdir.com/ml/java.jsr.166-concurrency/2005-04/msg00046.html

http://www.devwebsphere.com/devwebsphere/2005/06/dont_use_thread.html

UPDATE: Heinz Kabutz has written in a lot more depth about this problem in his newsletter: http://www.javaspecialists.eu/archive/Issue164.html
 

Tags :
Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Complexity makes for exponential maintenance costs?!

The title of this article isn't a fact, just a thought. However the thought comes from the experience of putting a very complex application into production and watching what happens. So long as there is stuff to do on the application, e.g. further releases adding more functionality, the maintenance costs are minimal, because you can plan the people who do the maintenance to do spend say 20% of their week on maintenance tasks while spending the rest of their time on building the new functionality.

But what happens when there is no more new functionality to build? What happens if suddenly there is no budget for new functionality? Well, you normally scale down the team, reducing head count to the minimum required to keep the knowledge about the application. You will probably put them part-time on other projects.

But now it gets hard. If the application is very complex or uses many differing technologies, it might not be possible to efficiently (in terms of reducing knowledge loss) reduce the size of the team below a specific size. If you application uses say a BPEL Engine, an Oracle Database, a Java Application Server, has a web front end and connects to several external business partners requiring knowledge of complex business rules, you probably already have the requirement to keep more than five people for maintenance. Mostly, this doesn't cause a problem, because those people can happily staff other projects. But if the application is so complicated that they are certain to forget parts of the system if they start working elsewhere, then you face a problem - knowledge loss is guaranteed if these people are not kept on the project for 100% of their time. Saying that, it's not even that simple, because the people might start forgetting parts of the system if they are not actively using / updateing / maintaining them. Just because you are on a project doesn't mean you are not forgetting how it works.

What can you do if you have such a complex project? First of all, from the start of development, ensure it is well documented, has good unit and regression tests. These will help reduce maintenance costs because it helps remind developers of the pitfalls in the system and checks that any changes they make don't introduce further bugs. But we know that we should do this on all projects anyway right? What else can you do? Well, the example I am thinking of used to have 50 people working on it. That was cut down to around 10, of which 5 were developers. If none of the developers are happy to work on anything else for more than 50% of the time because they would lose knowledge of the system, if you include server costs, an application manager and a few other comparatively small costs, you soon come to a budget of a million dollars a year just to keep your application in production. That cost is not related to number of bugs or number of technology updates (e.g. upgrading the application server in line with company policy to keep up with supported versions). That cost is just to keep the application running and have a safety net, in case a bug crops up (although it does of course include the cost to fix such bugs). Oh, and that price assumes there are other projects for your staff to go to - which might not be the case, if for example there is no budget, or if other projects are only interested in resources who are at least 75% available... If these problems apply, you suddenly need to double your maintenance budget because you have to keep these people on board 100% of their time!

Other options... Well if you compare this phase of a project to an earlier one where you have your staff doing 20% maintenance and 80% development. In this early project phase, maintenance time is optimum because the people are still working with the system and have all its nuances at the forefront of their minds. In the later phase (call it the maintenance phase where no new functionality is delivered), maintenance at 20% might actually cost the equivalent of 30% in the early phase, since the people are idling (working elsewhere) for 50% of their time and forgetting how the system works. So of their 50% availability, they only have 20% available (one day a week) in which to potentially build new functionality, should there be such requirements. Compare that to the earlier phase, a developer has 80% of their time to devote to new functionality (4 days a week) - they are four times as productive. But they are there for 100% of the time, so its costing you double compared to the later phase. In total, that makes them TWICE as efficient in terms of what is built per dollar!

Another problem with having staff working for only 50% on maintenance tasks is that they will get bored and want to move on. Bad morale is never a good thing.

Another option is to just ramp down regardless of knowledge loss and save money up front. That saved money can be put in a pot and be used to pay for any ramp up time needed if and when bugs or requirements for new functionality come your way (although it is unlikely that it will remain in that pot, since other more active projects are likely to use it). As a techy I would shy away from this approach, because experience has shown that ramping up a project at a later time almost guarantees that the newly built stuff will be badly integrated making the entire thing less maintainable, ensuring that maintenance costs in the future will be increased. Still it's your project, your choice. Since software decisions are often based on political decisions these days, you probably don't care if the project fails during a later maintenance stage, because it was successful while you were there (after all, you saved loads of money on maintenance back then!), and the person who took over after you left, must have screwed up because it now costs more to maintain or upgrade than when you were there. That sounds like good cause for another promotion for you! Just start praying now, that there are no bugs or change requests between the start of the maintenance phase where you slash the head count and the time you are no longer responsible for the project...

Side Note: The last paragraph reminds me of a project from some years back, where a project manager totally disregarded software quality and maintainability in order to get the first phase of the project (in fact entire platform) into production as fast as possible. He turned the whole project into a high profile one and span it all in his favour so that he delivered a great new platform on time and under budget guaranteeing his promotion into a role well above the previous one. By the time anyone realised that the platform was very expensive to maintain he was well out of sight and the person who took over spent a large proportion of their time defending the platform and trying to improve it. But from that place far above where seagulls fly, it was the new guy who looked bad, not the one who delivered a high profile project successfully... (note the reference to why directors are like seagulls, because they go around with their heads in the clouds crapping on the people below them).

Anyhow... it appears from the discussion above that there are three options when it comes to deciding how to put a very complex project into a maintenance phase.

  1. Do it traditionally, which is inefficient in terms of developers time
  2. Cut all staff to a bear minimum, and pray you are long gone before the crap hits the fan
  3. Don't do it at all! Keep developing, and justify it by showing that development is twice as efficient in doing so...


OK, well that last option does depend upon there actually being something to do. But most enterprise applications that I have worked on always went into the maintenance phase before they were completely finished. And of course more seriously, any software should always show a return on its investment - you can't just keep developing to the point that you are streaming live TV through your application because there was nothing else left to do! But I guess the important point is this: delay your maintenance phase until the ABSOLUTE last possible moment. If you intend on putting a project into a maintenance phase and then developing some other functionality in a year or twos time, it will be far more successful if you replan, keep your team running at 100% and bring the budgets for the new functionality forwards.

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!