<< July 2008 | Home | September 2008 >>

Reporting Stats

A few years ago when I was the integration architect in charge of my Global 500 customers EAI projects, the product that we used did not really offer any way to track the number of messages travelling through the system.

To get around this, we added a little library that recorded a small amount of information about each message. The idea was that every application (message flow) within the EAI system would be instrumented with this code and we would gain an oversight over which interfaces were the important ones and which ones made a return on their investment.

The sort of data we collected was:

  • Interface / Application / Process Name
  • Message Type
  • Message Size
  • Timestamp
  • Status (eg. IN, OUT, SUCCESS, FAIL)
  • Correlation Token (in case several messages belonged together)
  • Error (ie. details of an error)
  • Custom Data (specific to each interface / app / process)


There was probably more... it was a few years ago now :-)

Anyway, we soon discovered that there were applications that were not even being used, even though our customers had paid many tens of thousands of dollars for them (ok, I admit that isn't a lot of money in the IT world).

Since then Google has created its "Analytics" offering which generates some excellent reports for web site statistics. Indeed, maxant has always collected web stats for the websites it runs. It is very important to know your users / customers usage patterns, in order to be able to understand their real requirements and to improve the software over time.

Traditionally, stats might be gathered using some data warehousing techniques, trying to gather the data together from several existing data sources. In my experience, it is very simple to add a little library that reports stats data to a table or two. This has the advantage over traditional data warehousing, that you log exactly what you want, and not what happens to be available elsewhere. If you need the unique key of a row, log that, as opposed to the friendly name that might be logged elsewhere. You can use join statements when generating your reports to make them user friendly. Alternatively, you can just log both foreign keys and friendly names - making the database human readable, and providing accuracy in reports that need it. In one project we logged over 60 columns, allowing us to create around 30 reports (letting users drill down in detail where they needed it). Of course, if you log that much information, you need to estimate your data growth rate (about 20 Mb a day in our case) and define a suitable strategy for deleting or archiving old data, before you run out of disk space, or worse, start paying the data center for additional space when you don't really need it.

Also useful, is a switch, allowing you to shut off your reporting in case something goes wrong. Reporting should never interfere with the day to day running of your systems. If it turns out there is a bug in your reporting library, you don't want to have to wait for a hotfix to be deployed in your production environment, before you can continue processing data. Imagine your application is part of a sales system. The customer doesn't want to miss out on buying products just because the reporting might be out of whack for this month. The company CERTAINLY doesn't want to lose sales in favour of having a report that tells them they sold nothing for a week while waiting for the hotfix.

Statistics Reporting of software applications isn't limited to web or integration applications either. I have used it in rich client applications to track how the application is used, as well as server side to track for examples sales being made. Tracking data is also useful if it includes the inputs which the users made in order to generate the outputs. This can help in bug fixing and problem solving, since you will be able to recreate the issues because you know what the inputs were.

So, start reporting today! Then you can have a fun little project to generate dynamic reports - your managers and clients will love the pretty colourful graphs you can create! And that will help to keep the budget healthy for future work in your department :-)

It reminds me of when I worked in the aeronautical engineering world. There is a field called Computational Fluid Dynamics (CFD for short) which provides pretty pictures showing airflow over say a wing. It is often nicknamed Colours for Directors (also CFD), because directors love all the pretty colours and pictures, even though they might not fully appreciate the information lurking within the report.

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Spring Services Anywhere

I recently embarked on some Spring "re-training" (I haven't looked at it in a few years and it was time to get my skills up to scratch again). Since the last time I looked at Spring, I have worked on a couple of rich (thick) client projects where it was almost impossible to find people with both server skills and GUI skills (the market is quite good at the moment and people like me with all those skills get snapped up very quickly ).

Anyway, since it is hard to find strong GUI programmers who also know how to deploy an app server and start it, it becomes desirable to be able to run the services inside the same JVM as the GUI programmer is developing in. There is then no wasted time due to EJB configurations and all the crap that comes with server development.

About 18 months ago, I got around the problem by quickly implementing a clever service locator. It worked brilliantly, the only real problem being that we had a few bugs because people had developed code which worked when services were local but not when they were remote. The effort in fixing those bugs was however minimal.

On my current project, we are using a proprietary framework based on Inversion of Control and Dependency Injection. Based upon its configuration, a client can call services locally or remotely too. If calling services remotely, they are invoked inside a stateless session bean which uses reflection and proxies in order to load the service and call its method. Transaction and security inteceptors are also in place ensuring that the services are called in the correct context. It is all very clever, but sadly proprietary (although at least we have the source code!).

Anyway, so as I am reading up on Spring, I am reminded how it too is intended to provide services independently of the environment in which it is running (that is, whether it is running in a stand alone application, inside a web container or an EJB container).

However, there is a marked difference compared to my Service Locator solution and the proprietary solution of my current project. In Spring, services are not automatically "remoted". To move services from a standalone application to run inside an EJB container, you need to write code which sits between the client and those services to allow them to be called.

In EJB 3.0, a stateless session bean is simply a POJO. If it is not deployed in a server, a standalone application which has that POJO on its classpath can call a method in the POJO (admittedly without security or transaction functionality).

In Spring, you deploy your service to a Web or EJB Container and that container can use the services, but the client is no longer able to call those services. At the best, you can "remote" your service so that it can be called through its interface, however, it will live inside the web container. If you want the client to be able to call your service that is now in the EJB container, you need to write an EJB with the correct interface to expose that services functionality.

Now, is that a problem? What's wrong with just using the web container or indeed just a web server? Ahhh... its not capable of handling transactions, I hear you say. True, but Spring provides mechanisms for applying a transactional context to your services. You can even use the transaction manager from your application server, so it will provide the same transactional functionality as provided to EJBs (for example XA compliant and having the ability to run across nodes in a cluster).

OK, OK, the web container doesn't provide security to service methods! Well, also not true, since you can use declarative and programmatic security from the web container to ensure the user can only use functionality for which they are authorised.

OK, but the web container isn't scalable/reliable/robust! Also not true, just like the EJB container, it has the ability to synchronise state across a cluster. And running in a cluster also means it is scalable, reliable and robust.

So, you might now start to agree, that actually, there is no need to run in an application server. Everything you do day in and day out can now be handled by Spring in the web container, ie. by a friendly cluster-aware web server like Apache Tomcat...

I wouldn't disagree with that statement. However, here are some reasons to run inside a full application server:

  • You need a JTA Transaction manager capable of propagating transactions across nodes in a cluster,
  • The enterprise architecture department in charge of specifying the environment you run in is also capable of specifying that services must run in the EJB layer,
  • You need to receive JMS messages in message beans
  • You need to use Timer Beans
  • You need a clustered JNDI tree as provided by your application server
  • You need access to resources which your application server can provide and manage (eg. JCA)

If you don't have any of these constraints, an you are committed to using Spring, then I currently do not see many reasons to run in anything other than a Java EE compliant Web Container. If you can think of other reasons to run in a full application server, please leave comments below.

Tags : ,
Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Secure Remoting with Spring and JBoss

If you are faced with having to write a rich client application in a multi-tier Java EE environment, you will typically connect to the application server over RMI. In theory, you are meant to use the servers Application Client Container and deploy your application as a client in that container. You probably won't do that though, because the client container is unfriendly for many reasons:

  1. As an example, the WebSphere 6.1 Client Container is a 200 megabyte install,
  2. Client Containers tend to be started as batch commands which set up the environment in which your application will run. If you however have an application that is meant to be started with a sexy launcher, as is the case with Eclipse RCP applications, you will struggle to get the environment created properly by the launcher, and its not supported by the vendor anyway,
  3. If you need to connect to the server securely (ie. so that serverside you have a valid security context allowing you to authorise users to call given services), then I personally have never been able to get the security callback mechanism to work. Theoretically you can tell the container to call your code at the point which it logs on to the server in order to get the credentials (eg. you can pop up a little login window),

For these reasons, I have never ever used a client container in a production environment. Instead I have repeatedly gone to the trouble of getting the client environment fit so that it can call the server over RMI with a security context. The problem is not only that this is painful, but that it is unsupported by the App Server vendor, and is almost guaranteed to break with the next release of the App Server, requiring further pain from a consultant in order to get be able to migrate to the next App Server version. Additionally, setting the environment for the Initial Context in a client, to connect to the server and find objects in the JNDI tree can be difficult and is often poorly documented (after all, you are meant to do it from inside the client container where you do not need to provide the environment yourself).

However, there is another way! In a previous article, I wrote about how I got an applet to send information back to the server by implementing a custom protocol containing serialised objects. In hindsight I discovered Hessian from Caucho which does a similar thing. Recently I have been looking into Spring which supports Hessian (Custom Binary over HTTP), Burlap (XML over HTTP) and its own kind of remoting, namely Java Serialisation over HTTP.

With little effort I was able to find out how to secure the HTTP connection with the server, meaning that I am now free to write an arbitrary rich client who can call services in a Java EE environment without the use of a Client Container, and without a non-supported RMI environment that is likely to not work after a server version migration. Woo Hoo! How? Easy, keep reading...

First, lets start with a little diagram:



This article won't go into detail about how to create an RCP application or plugin, and will simply call the remote service from a simple Java application. The steps involved in getting to where we are going are:

  1. Add a Datasource to JBoss
  2. Add Security to JBoss (using the datasource, although LDAP would be an option too)
  3. Create a Web Application containing the service implementation
  4. Create a client capable of calling the remote service

1) Add a datasource to JBoss

For this article, JBoss 4.2.2 was used. To add a datasource, create a configuration file:

    <JBoss-Home>/server/<instance>/deploy/mysql-ds.xml

where <instance> was "default" in the case of this example. This example uses MySQL 4.1. This configuration file should contain the following:


A restart of JBoss will make this datasource available in JNDI under the name java:/jdbc/MySqlDS.

2. Add Security to JBoss

Add the following application policy to:

    <JBoss-Home>/server/<instance>/conf/login-config.xml



Before it can work, you will need to add two tables to the database. The following is some SQL to do that:



3) Create Web App

To create the web application you need the following files:



web.xml contains:


In the above web.xml, I have configured a Spring ContextLoaderListener which uses the contextConfigLocation parameter to load the Spring configuration. A servlet has also been configured for the URL pattern "remoting" and this URL pattern has also been secured, requiring basic authentication in order to be requested.

remoting-servlet.xml contains:


The above remoting-servlet.xml is firstly named after the dispatcher servlet which was defined in web.xml. Secondly, it defines the service exporter and our service that we wish to make remote, namely "TestService". What is missing from the above is the definition of the "testService" bean. That definition is located under the applicationContext.xml:
That bean definition could equally have actually been included in remoting-servlet.xml.
Finally, the jboss-web.xml file contains the link between the security policy that was added to login-config.xml, and the web application:


Next, on the web app classpath, we need to have two classes, namely the service interface and its implementation.
The web application now contains everything it needs to go. Restart JBoss and deploy the web application to it.

4) Create Client
All that is now left is to create a client that can call the remote service. Create a project with the following files:


To start with, the applicationContext.xml is a Spring configuration and looks like this:

In the Spring configuration above, the remote service is defined in terms of its URL and its interface class. The "basicAuthenticationInvokerRequestExecutor" bean simply tells the "HttpInvokerProxyFactoryBean" to use a special InvokerRequestExecutor which is capable of logging into the service. The BasicAuthenticationInvokerRequestExecutor basically comes from a Spring Forum, and is shown in its modified form next:

This class uses the Apache Commons HttpClient to use basic authentication when it logs into the web application when it calls the remote service.

Finally, the code which calls the remote service:
On thing missing in the above description, is that the client needs a reference to the service interface. In this example, built in Eclipse 3.4, the client project simply has a reference to the web application project containing the service interface. In reality you would want to have this interface in a client library which both the web application and the client can refer to.

Now everything is in place to call services on a remote server from a client, securely, and independently of application server Client Container. I chose to use Spring HTTP Remoting with standard Java serialisation (as opposed to Hessian or Burlap, which have their own problems due to restrictive serialisation), which locks the client and server to compatible Java versions. However, Java serialisation has not changed much in a while and I have successfully tested a client running in Sun Java 1.3, calling a Sun Java 1.6 server!

That is the end of this article, I hope it works for you, and gives you an insight into how to securely call services in a Java EE application server without the need to use the client container. Good Luck!

Tags : , ,
Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!