The Kitchen in the Zoo - howto tag

Non-blocking (NIO) Server Push and Servlet 3

Sun, 05 Jun 2011 18:40:00 GMT

In my previous blog posting, I wrote about what I would expect node.js to do in order to become mature. I introduced the idea of having a framework which lets you define a protocol and some handlers in order to let the developer concentrate on writing useful business software, rather than technical code, in a very similar manner to which Java EE does. And through that posting, I came to learn about a thing called Comet. I had stated that using a non-blocking server wouldn't really be any more useful than a blocking one in a typical web application (HTTP), and so I created an example based on my own protocol and a VOIP server, for streaming binary data to many concurrently connected clients.

I have now read up on Comet and come to realise there is indeed a good case for having a non-blocking server in the web. That case is pushing data back to the client, like for continuously publishing latest stock prices. While this example could be solved using polling, true Comet uses long-polling or even better, full on push. A great introduction I read was here. The idea is that the client makes a call to the server and instead of the server returning data immediately, it keeps the connection open and returns data at some time in the future, potentially many times. This is not a new idea - the term Comet seems to have been invented in about 2006 and the article I refer to above is from 2009. I think I've arrived at this party very late :-)

My new found case of server push for non-blocking HTTP, and a strong curiosity drove me to knuckle down and start coding. Before long, I had a rough implementation of the HTTP protocol for my little framework, and I was able to write an app for my server using handlers that subclassed the HttpHandler, which was for all intents and purposes, a Servlet.

To get Comet push to work properly, you get the client to "log in" and register itself with the server. In my demo, I didn't check authorisations against a database, like you might do for a real app, but I had the concept of a channel, to which any browser client could subscribe. During this login, the client says which channel it wants to subscribe to, and the server adds the clients non-blocking connection to its model. The server responds using chunked transfer encoding, because that way, the connection stays open, and you don't need to state up front how much data you will send back. At some time in the future, when someone publishes something, the server can use the connection which is still open contained in its model, to push that published data back to the subscribed client, by sending another chunk of data.

The server implementation wasn't too hard, but the client posed a few problems, until I realised that the data was arriving in the ajax client with a ready state of 3, rather than the more usual 4. The ajax client's onreadystatechange callback function was also given every byte of data in it's responseText, rather than just the new stuff, so I had some fiddling around until I could get the browser to just append the new stuff to the innerHTML attribute of a div on my page. Anyway, after just a few hours, I had an app that worked quite well. But it wasn't entirely satisfactory, partly because as I stated in the previous posting, the server is still a little buggy, especially when the client drops a connection, because for example the browser page is closed. I had also ended up implementing the HTTP protocol for my framework, which seemed to be reinventing the wheel - servlet technology does all this stuff already, and much better than I can hope to do it. One of the reasons that I don't like node.js is that everything is being reinvented.

So, like the article which I referenced above indicated, version 3.0 of Servlets should be able to handle Comet. I downloaded Tomcat 7.0, which has a Servlet 3.0 container, and I ported my app code to proper Servlets. It took a while to work out exactly how to use the new asynchronous parts of servlets because there aren't all that many accurate tutorials out there. The servlet specs (JSR 315) helped a lot. Once I cracked how to use the async stuff properly, I had a really really satisfying solution to my push requirements.

The first step, was to reconfigure Tomcat so that it uses non-blocking (NIO) for its connector protocol. The point of this is that I want to keep a connection open to the client in order to push data to it. I can't rely on the one-thread-per-request paradigm, because context switching and thread memory requirements are likely to be a performance killer. In Tomcat's server.xml file, I configured the connector's protocol:

	<!-- NIO HTTP/1.1 connector -->    
    <Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol" 
               connectionTimeout="20000" 
               redirectPort="8443" />

rather than the normal:

    <Connector port="8080" protocol="HTTP/1.1" 
               connectionTimeout="20000" 
               redirectPort="8443" />

All you need to do to get Tomcat to turn into an NIO server is change the protocol attribute to the slightly longer class name.

The second step, was to create two servlets. The first LoginServlet handles the client "logging in" and subscribing to a channel. That servlet looks like this:

/*  
 * Copyright (c) 2011 Ant Kutschera
 * 
 * This file is part of Ant Kutschera's blog, 
 * http://blog.maxant.co.uk
 * 
 * This is free software: you can redistribute
 * it and/or modify it under the terms of the
 * Lesser GNU General Public License as published by
 * the Free Software Foundation, either version 3 of
 * the License, or (at your option) any later version.
 * 
 * It is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied
 * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
 * PURPOSE.  See the Lesser GNU General Public License for
 * more details. 
 * 
 * You should have received a copy of the
 * Lesser GNU General Public License along with this software.
 * If not, see http://www.gnu.org/licenses/.
 */
package ch.maxant.blog.nio.servlet3;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;

import javax.servlet.AsyncContext;
import javax.servlet.ServletContext;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import ch.maxant.blog.nio.servlet3.model.Subscriber;

@WebServlet(name = "loginServlet", urlPatterns = { "/login" }, asyncSupported = true)
public class LoginServlet extends HttpServlet {

	public static final String CLIENTS = "ch.maxant.blog.nio.servlet3.clients";

	private static final long serialVersionUID = 1L;

	public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {

		// dont set the content length in the response, and we will end up with chunked 
		// encoding so that a) we can keep the connection open to the client, and b) send
		// updates to the client as chunks.
		
		// *********************
		// we use asyncSupported=true on the annotation for two reasons. first of all, 
		// it means the connection to the client isn't closed by the container.  second 
		// it means that we can pass the asyncContext to another thread (eg the publisher) 
		// which can then send data back to that open connection.
		// so that we dont require a thread per client, we also use NIO, configured in the 
		// connector of our app server (eg tomcat)
		// *********************

		// what channel does the user want to subscribe to?  
		// for production we would need to check authorisations here!
		String channel = request.getParameter("channel");

		// ok, get an async context which we can pass to another thread
		final AsyncContext aCtx = request.startAsync(request, response);

		// a little longer than default, to give us time to test.
		// TODO if we use a heartbeat, then time that to pulse at a similar rate
		aCtx.setTimeout(20000L); 

		// create a data object for this new subscription
		Subscriber subscriber = new Subscriber(aCtx, channel);

		// get the application scope so that we can add our data to the model
		ServletContext appScope = request.getServletContext();

		// fetch the model from the app scope
		@SuppressWarnings("unchecked")
		Map

The inline comments describe most of the choices I made. I added a listener to the context so that we get an event in the case of a disconnected client, so that we can tidy up our model - the full code is in the ZIP at the end of this article. Importantly, this servlet does no async processing itself. It simply prepares the request and response objects for access at some time in the future. We stick them (via the async context) into a model which is in application scope. That model can then be used by the servlet which receives a request to publish something to a given channel:

/*  
 * Copyright (c) 2011 Ant Kutschera
 * 
 * This file is part of Ant Kutschera's blog, 
 * http://blog.maxant.co.uk
 * 
 * This is free software: you can redistribute
 * it and/or modify it under the terms of the
 * Lesser GNU General Public License as published by
 * the Free Software Foundation, either version 3 of
 * the License, or (at your option) any later version.
 * 
 * It is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied
 * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
 * PURPOSE.  See the Lesser GNU General Public License for
 * more details. 
 * 
 * You should have received a copy of the
 * Lesser GNU General Public License along with this software.
 * If not, see http://www.gnu.org/licenses/.
 */
package ch.maxant.blog.nio.servlet3;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Map;

import javax.servlet.AsyncContext;
import javax.servlet.ServletContext;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import ch.maxant.blog.nio.servlet3.model.Subscriber;

@WebServlet(name = "publishServlet", urlPatterns = { "/publish" }, asyncSupported = true)
public class PublishServlet extends HttpServlet {

	private static final long serialVersionUID = 1L;

	public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {

		// *************************
		// this servlet simply spawns a thread to send its message to all subscribers.
		// this servlet keeps the connection to its client open long enough to tell it 
		// that it has published to all subscribers.
		// *************************
		
		// add a pipe character, so that the client knows from where the newest model has started.
		// if messages are published really quick, its possible that the client gets two at
		// once, and we dont want it to be confused!  these messages also arrive at the 
		// ajax client in readyState 3, where the responseText contains everything since login,
		// rather than just the latest chunk.  so, the client needs a way to work out the 
		// latest part of the message, containing the newest version of the model it should 
		// work with.  might be better to return XML or JSON here!
		final String msg = "|" + request.getParameter("message") + " " + new Date();

		// to which channel should it publish?  in prod, we would check authorisations here too!
		final String channel = request.getParameter("channel");

		// get the application scoped model, and copy the list of subscribers, so that the 
		// long running task of publishing doesnt interfere with new logins
		ServletContext appScope = request.getServletContext();
		@SuppressWarnings("unchecked")
		final Map

Again, there are plenty of comments in the code. In this servlet, we actually do some (potentially) long running task. In many online examples of Servlet 3.0 Async Support, they show handing off work to an Executor. The async context provides the ideal way to do this via the container though, using its start(Runnable) method. The container implementation then descides how to handle the task, rather than the developer having to worry about spawning threads, which on app servers like WebSphere is illegal and leads to errors. In true Java EE fashion, the developer can concentrate on business code, rather than technicalities.

Something else which might be important in the above code, is that the publishing is done on a different thread. Imagine publishing data to ten thousand clients. In order to finish within a second, each push it going to have to complete in less than a tenth of a millisecond. That means doing something useful like a database lookup is not going to be feasible. On a non-blocking server, we can't afford to take a second to do something, and many would argue a second is eternity in such an environment. The ability to hand off such work to a different thread is invaluable, and sadly, something node.js cannot currently do, although you can hand off the task to a different process, albeit potentially messier than that shown here.

Now, we just need to create a client to subscribe to the server. This client is an ajax request object which is created when the HTML is loaded which runs some JavaScript in a library I have written. The HTML looks like this:

<html>
<head>
	<script language="Javascript" type="text/javascript" src="push_client.js"></script>
</head>
<body>
<p>Boo!</p>
<div id="myDiv"></div>
</body>
<script language="Javascript" type="text/javascript">

function callback(model){
	//simply append the model to a div, for demo purposes
	var myDiv = document.getElementById("myDiv");
	myDiv.innerHTML = myDiv.innerHTML + "<br>" + model;
}

new PushClient("myChannel", callback).login();

</script>
</html>

As you can see, it simply needs to define a callback function which will handle each message published from the server. The published message could be text, XML or JSON - the publisher chooses. The JavaScript in that library is a little more complicated, but basically creates an XHR requester which sends a request to the login servlet. Any data it receives from the server, it parses, and returns the newest part of the data back to the callback.

/*  
 * Copyright (c) 2011 Ant Kutschera
 * 
 * This file is part of Ant Kutschera's blog, 
 * http://blog.maxant.co.uk
 * 
 * This is free software: you can redistribute
 * it and/or modify it under the terms of the
 * Lesser GNU General Public License as published by
 * the Free Software Foundation, either version 3 of
 * the License, or (at your option) any later version.
 * 
 * It is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied
 * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
 * PURPOSE.  See the Lesser GNU General Public License for
 * more details. 
 * 
 * You should have received a copy of the
 * Lesser GNU General Public License along with this software.
 * If not, see http://www.gnu.org/licenses/.
 */

function PushClient(ch, m){

	this.channel = ch;
	this.ajax = getAjaxClient();
	this.onMessage = m;

	// stick a reference to "this" into the ajax client, so that the handleMessage 
	// function can access the push client - its "this" is an XMLHttpRequest object
	// rather than the push client, coz thats how javascript works!
	this.ajax.pushClient = this;
	
	function getAjaxClient(){
		/*
		 * Gets the ajax client
		 * http://en.wikipedia.org/wiki/XMLHttpRequest
		 * http://www.w3.org/TR/XMLHttpRequest/#responsetext
		 */
	    var client = null;
	    try{
			// Firefox, Opera 8.0+, Safari
			client = new XMLHttpRequest();
		}catch (e){
			// Internet Explorer
			try{
				client = new ActiveXObject("Msxml2.XMLHTTP");
			}catch (e){
				client = new ActiveXObject("Microsoft.XMLHTTP");
			}
		}
		return client;
	};
	
	/** 
	 * pass in a callback and a channel.  
	 * the callback should take a string, 
	 * which is the latest version of the model 
	 */
	PushClient.prototype.login = function(){

		try{
			var params = escape("channel") + "=" + escape(this.channel);
			var url = "login?" + params;
			this.ajax.onreadystatechange = handleMessage;
			this.ajax.open("GET",url,true); //true means async, which is the safest way to do it
			
			// hint to the browser and server, that we are doing something long running
			// initial tests only seemed to work with this - dont know, perhaps now it 
			// works without it?
			this.ajax.setRequestHeader("Connection", "Keep-Alive");
			this.ajax.setRequestHeader("Keep-Alive", "timeout=999, max=99");
			this.ajax.setRequestHeader("Transfer-Encoding", "chunked");
			
			//send the GET request to the server
			this.ajax.send(null);
		}catch(e){
			alert(e);
		}
	};

	function handleMessage() {
		//states are:
		//	0 (Uninitialized)	The object has been created, but not initialized (the open method has not been called).
		//	1 (Open)	The object has been created, but the send method has not been called.
		//	2 (Sent)	The send method has been called. responseText is not available. responseBody is not available.
		//	3 (Receiving)	Some data has been received. responseText is not available. responseBody is not available.
		//	4 (Loaded)
		try{
			if(this.readyState == 0){
				//this.pushClient.onMessage("0/-/-");
			}else if (this.readyState == 1){
				//this.pushClient.onMessage("1/-/-");
			}else if (this.readyState == 2){
				//this.pushClient.onMessage("2/-/-");
			}else if (this.readyState == 3){
				//for chunked encoding, we get the newest version of the entire response here, 
				//rather than in readyState 4, which is more usual.
				if (this.status == 200){
					this.pushClient.onMessage("3/200/" + this.responseText.substring(this.responseText.lastIndexOf("|")));
				}else{
					this.pushClient.onMessage("3/" + this.status + "/-");
				}
			}else if (this.readyState == 4){
				if (this.status == 200){
					
					//the connection is now closed.
					
					this.pushClient.onMessage("4/200/" + this.responseText.substring(this.responseText.lastIndexOf("|")));

					//start again - we were just disconnected!
					this.pushClient.login();

				}else{
					this.pushClient.onMessage("4/" + this.status + "/-");
				}
			}
		}catch(e){
			alert(e);
		}
	};
}

The important parts here are that we need to listen for events on both ready state 3 and 4, rather than the usual 4. While the connection is kept open, the client only receives data in ready state 3, and everytime it receives a chunk, the ajax.responseText attribute contains all chunks since login, rather than just the newest chunk. This could be bad, if the connection receives tons of data - eventually the browser will run out of memory! You could measure the number of bytes sent to any client on the server, and when its above a given threshold, force the client to disconnect by ending the stream (call the complete() method on the async context of the relevant client just after publishing a message to it). The client above automatically logs itself back into the server when the server disconnects.

Instead of the reconnect solution for dropped / timed-out connections (which is very similar to long polling) we could add a heartbeat which the server sends to each subscribed client. The heartbeat period would need to be slightly less than the timeout. The exact details could get messy - do you consider sending a heartbeat every second, but only do it to clients who need it? Or do you send it to every client, say every 25 seconds, if the timeout is say 30 seconds? You could use performance tuning to determine if that is better than the reconnecting I have shown above. Then again, a heartbeat is good for culling closed connections, because it tests the connection every so often, and gets an exception if the push fails. And then again, the container informs the listener that we added to the login async context if a disconnect occurs too, so perhaps we don't need a heartbeat - you decide :-)

Now, we need a way to publish data - that's easy - I just type the following URL into the browser, and it sends a GET request to the publish servlet:

http://localhost:8080/nio-servlet3/publish?channel=myChannel&message=javaIsAwesome

Keep refreshing the browser window with that protocol, and the other window almost instantly gets updates, showing the latest message at the bottom. I tested it using Firefox as the subscriber, and Chrome as the publisher.

I haven't checked out scalability, because I have assumed that Tomcat's NIO connector has been well tested and performs well. I'll let someone else play with the scalability of Servlet 3.0 for a push solution. This article is about showing how easy it is to implement Comet push, using Java EE Servlets. Note, it used to be easy too, because servers like Jetty and Tomcat and others provided bespoke Comet interfaces, but now, with the advent of Servlet 3.0, there is a standardised way to do it.

There are plenty of profesional and open source solutions which do what I have done in this article. See this article, which lists many, whackiest of all, APE - another project which puts JavaScript on the server!? Well, better than PHP I guess :-)

The complete code for this demo is downloadable here.

Copyright © 2011 Ant Kutschera

Social Bookmarks :

Node JS and Server side Java Script

Sat, 05 Mar 2011 21:36:00 GMT

Let's start right at the beginning. Bear with me, it might get long...

The following snippet of Java code could be used to create a server which receives TCP/IP requests:

class Server implements Runnable {
    public void run() {
        try {
            ServerSocket ss = new ServerSocket(PORT);
            while (!Thread.interrupted())
                Socket s = ss.accept();
                s.getInputStream(); //read from this
                s.getOutputStream(); //write to this 
        } catch (IOException ex) { /* ... */ }
    }
}

This code runs as far as the line with ss.accept(), which blocks until an incoming request is received. The accept method then returns and you have access to the input and output streams in order to communicate with the client.

There is one issue with this code. Think about multiple requests coming in at the same time. You are dedicated to completing the first request before making the next call to the accept method. Why? Because the accept method blocks. If you decided you would read a chunk off the input stream of the first connection, and then be kind to the next connection and accept it and handle its first chunk before continuing with the original (first) connection, you would have a problem, because the accept method blocks. If there were no second request, you wouldn't be able to finish off the first request, because the JVM blocks on that accept method. So, you must handle an incoming request in its entirety, before accepting a second incoming request.

This isn't so bad, because you can create the ServerSocket with an additional parameter, called the backlog, which tells it how many requests to queue up before refusing further connections. While you are busy handling the first request, subsequent requests are simply queued up.

This strategy would work, although it's not really efficient. If you have a multicore CPU, you will only be doing work on one core. It would be better to have more threads, so that the load can be balanced across the cores (watch out, this is JVM and OS dependent!).

A more typical multi-threaded server gets built like this:

class Server implements Runnable {
    public void run() {
        try {
            ServerSocket ss = new ServerSocket(PORT);
            while (!Thread.interrupted())
                new Thread(new Handler(ss.accept())).start();
                // one thread per socket connection every thread 
                // created this way will essentially block for I/O
        } catch (IOException ex) { /* ... */ }
    }
}

The above code hands off each incoming request to a new thread, allowing the main thread to handle new incoming requests, while spawned threads handle individual requests. This code also balances the load across CPU cores, where the JVM and OS allow it. Ideally, we probably wouldn't create a thread per new request, but rather hand off the request to a thread pool executor (see the java.util.concurrent package). On the other hand, there are times when a thread per request is required. If the conversation between server and client is longer lasting (rather than a simple HTTP request that is typically serviced in anything from milliseconds to seconds), then the socket can stay open. An example of when this is required are things like chat servers, or VOIP, or anything else where a continual conversation is required. But in such situations, the above code, even though it is multi-threaded, has it's limits. Those limits are actually because of the threads! Consider the following code:

public class MaxThreadTest {

    static int numLive = 0;
	
    public static void main(String[] args) {
        while(true){
            new Thread(new Runnable(){
                public void run() {
                    numLive++;
					
                    System.out.println("running " + Thread.currentThread().getName() + " " + numLive);
                    try {
                        Thread.sleep(10000L);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }

                    numLive--;
                }
            }).start();
        }
    }
}

This code creates a bunch of threads, until the process crashes. With 64 MB heap size, it crashed (out of memory) after around 4000 threads, while testing on my Windows XP Thinkpad laptop. I upped the heap size to 256 MB and Eclipse crashed while in debug mode... I started the process from the command line and managed to open 5092 threads, but it was unstable and unresponsive. Interestingly, I upped the heap size to 1 GB, and then I could only open 2658 threads... This shows, I don't really understand the OS or JVM at this level! Anyway, if we were writing a system to handle a million simultaneous conversations, we would probably need over two hundred servers. But theoretically, we could reduce our costs to less than 10% of that, because we are allowed to open just over 65,000 threads per server (well, say 63,000 by the time we account for all the ports used by the OS and other processes). We could theoretically get away with just having 16 servers per million simultaneous connections.

The way to do this is, is to use non-blocking I/O. Since Java 1.4 (around 2002?), the java.nio package has been around to help us. With it, you can create a system which handles many simultaneous incoming requests using just one thread. The way it works is roughly by registering with the OS to get events when something happens, for example when a new request is accepted, or when one of the clients sends data over the wire.

With this API, we can create a server, which is, sadly, a little more complicated than those above, but which handles lots and lots of sockets all from one thread:

public class NonBlockingServer2 {

	public static void main(String[] args) throws IOException {
		System.out.println("Starting NIO server...");
		Charset charset = Charset.forName("UTF-8");
		CharsetDecoder decoder = charset.newDecoder();
		CharsetEncoder encoder = charset.newEncoder();
		
		ByteBuffer buffer = ByteBuffer.allocate(512);

		Selector selector = Selector.open();
		ServerSocketChannel server = ServerSocketChannel.open();
		server.socket().bind(new InetSocketAddress(30032));
		server.configureBlocking(false);
		SelectionKey serverkey = server.register(selector, SelectionKey.OP_ACCEPT);

		boolean quit = false;
		while(!quit) {
			selector.select(); //blocks until something arrives, of type OP_ACCEPT
			Set keys = selector.selectedKeys();
			for (SelectionKey key : keys) {
				if (key == serverkey) {
					if (key.isAcceptable()) {
						SocketChannel client = server.accept();
						if(client != null){ //can be null if theres no pending connection
							client.configureBlocking(false);
							SelectionKey clientkey = client.register(selector,
									SelectionKey.OP_READ); //register for the read event
							numConns++;
						}
					}
				} else {
					SocketChannel client = (SocketChannel) key.channel();
					if (!key.isReadable()){
						continue;
					}
					int bytesread = client.read(buffer);
					if (bytesread == -1) {
						//whens this happen?
						key.cancel();
						client.close();
						continue;
					}
					buffer.flip();
					String request = decoder.decode(buffer).toString();
					buffer.clear();
					
					if (request.trim().equals("quit")) {
						client.write(encoder.encode(CharBuffer.wrap("Bye.")));
						key.cancel();
						client.close();
					}else if (request.trim().equals("hello")) {
						String id = UUID.randomUUID().toString();
						key.attach(id);
						String response = id + "\r";
						client.write(encoder.encode(CharBuffer.wrap(response)));
					}else if (request.trim().equals("time")) {
						numTimeRequests++;
						String response = "hi " + key.attachment() + " the time here is " + new Date() + "\r";
						client.write(encoder.encode(CharBuffer.wrap(response)));
					}
				}
			}
		}
		System.out.println("done");
	}
}

The above code is based on that found here. By reducing the number of threads being used, and not blocking, but rather relying on the OS to tell us when something is up, we can handle many more requests. I tested some code very similar to this to see how many connections I could handle. Windows XP proved its high reliability, when reproducibly and consistently, more than 12,000 connections lead to blue screens of death! Time to move to Linux (Fedora Core). I had no problems creating 64,000 clients all simultaneously connected to my server. Let me re-prase... I didn't have problems having the clients simply connect and keep the connection open, but getting the server to also handle just 100 requests a second caused problems. Now 100 requests a second on a web server, on hardware which was a 5 year old cheap Dell laptop, sounds quite impressive to me. But on a server with 64,000 concurrent connections, that means each client making a request every ten minutes! Not very good for a VOIP application... The connection speeds also slowed down from around 3 milliseconds with 500 concurrent connections, down to 100 milliseconds with 60,000 concurrent connections.

So, perhaps I better get to the point of this posting? A few days ago, I read about Voxer, and Node.js on The Register. I had difficulty with this article. Why would anyone want to build a framework for Javascript on the server? I have developed plenty of rich clients, and have the experience to understand how to do rich client development. I have also developed plenty of rich internet apps (RIA), which use Javascript, and I can only say, it's not the best. I'm not some script kiddie or script hacker who doesn't know how to design Javascript code, and I understand the problems of Javascript development well. And I have developed lots and lots of server side code, mostly in Java and appreciate where Java out punches Javascript.

It seems to me, that the developers of Node.js, and those following it and using it, don't understand server development. While writing in Javascript might initially be quicker, the lack of tools and libraries in comparison to Java make it a non-competition in my opinion.

If I were a venture capitalist, and knew my money was being spent on application development based on newly developed frameworks, instead of extremely mature technologies, when the mature technologies suffice (as shown with the non-blocking server code above), I would flip out and can the project.

Maybe though, this is why I have never worked at a start up!

To wrap up, let's consider a few other points. Before anyone says that the performance of my example server was poor because it's just Java which is slow, let me comment. First of all, Java will always be faster than Javascript. Secondly, using top to monitor the server, I noticed that 50% of the CPU time was spent by the OS working out what events to throw, rather than Java handling those requests.

In the above server, everything runs on one thread. To improve performance, once a request comes in, it could be handed off to a thread pool to respond. This would help balance load across multiple cores, which is definitely required to make the server production ready.

While I'm at it, here is a quote from Node JS's home page:

"But what about multiple-processor concurrency? Aren't threads necessary to scale programs to multi-core computers? Processes are necessary to scale to multi-core computers, not memory-sharing threads. The fundamentals of scalable systems are fast networking and non-blocking design—the rest is message passing. In future versions, Node will be able to fork new processes (using the Web Workers API ) which fits well into the current design."

Actually, I'm not so sure... Java on Linux can spread threads across cores, so individual processes are not actually required. And the above statement just proves that Node JS is not mature for building really professional systems - I mean come on, no threading support?!

So, in the interests of completion, here is the client app I used to connect to the server:

public class Client {

	private static final int NUM_CLIENTS = 3000;

	static Timer serverCallingTimer = new Timer("servercaller", false);
	
	static Random random = new Random();

	/**
	 * this client is asynchronous, because it does not wait for a full response before 
	 * opening the next socket.
	 */
	public static void main(String[] args) throws UnknownHostException, IOException, InterruptedException {
		
		final InetSocketAddress endpoint = new InetSocketAddress("192.168.1.103", 30032);
		System.out.println(new SimpleDateFormat("HH:mm:ss.SSS").format(new Date()) + " Starting async client");

		long start = System.nanoTime();
		for(int i = 0; i < NUM_CLIENTS; i++){
			startConversation(endpoint);
		}

		System.out.println(new SimpleDateFormat("HH:mm:ss.SSS")
				.format(new Date())
				+ "Done, averaging "
				+ ((System.nanoTime() - start) / 1000000.0 / NUM_CLIENTS)
				+ "ms per call");
	}

	protected static void startConversation(InetSocketAddress endpoint) throws IOException {
		final Socket s = new Socket();
		s.connect(endpoint, 0/*no timeout*/);
		s.getOutputStream().write(("hello\r").getBytes("UTF-8")); //protocol dictates \r is end of command
		s.getOutputStream().flush();

		//read response
		String str = readResponse(s);
		System.out.println("New Client: Session ID " + str);

		//send a request at regular intervals, keeping the same socket! eg VOIP
		//we cannot use this thread, its the main one which created the socket
		//simply create another task to be carried out by the scheduler at a later time

		//the interval below is 4 minutes, otherwise the server gets REALLY slow handling 
		//so many requests.  This is equivalent to ~260 reqs/sec
		
		serverCallingTimer.scheduleAtFixedRate(
				new ConversationContainer(s, str), 
				random.nextInt(240000/*in the next 4 mins*/), 
				240000L/*every 4 mins*/);
	}
	
	private static String readResponse(Socket s) throws IOException {
		InputStream is = s.getInputStream();
		int curr = -1;
		ByteArrayOutputStream baos = new ByteArrayOutputStream();
		while((curr = is.read()) != -1){
			if(curr == 13) break; //protocol dictates a new line is the end of a response
			baos.write(curr);
		}
		return baos.toString("UTF-8");
	}

	private static class ConversationContainer extends TimerTask {
		Socket s;
		String id;
		public ConversationContainer(Socket s, String id){
			this.s = s;
			this.id = id;
		}
		
		@Override
		public void run() {
			try {
				s.getOutputStream().write("time\r".getBytes("UTF-8")); //protocol dictates \r is end of command
				s.getOutputStream().flush();

				String response = readResponse(s);
				
				if(random.nextInt(1000) % 1000 == 0){
					//we dont want to log everything, because it will kill our server!
					System.out.println(id + " - server time is '" + response + "'");
				}
				
			} catch (Exception e) {
				e.printStackTrace();
			}
		}
	}
}

Social Bookmarks :

Dynamic Mock Testing

Wed, 03 Nov 2010 19:45:00 GMT

Have you ever had to create a mock object in which most methods do nothing and are not called, but in others something useful needs to be done?

EasyMock has some newish functionality to let you stub individual methods. But before I had heard about that, I had built a little framework (one base class) for creating mock objects which stubs those methods you want to stub, as well as logging every call made to the classes being mocked.

It works like this: you choose a class which you need to mock, for example a service class called FooService, and you create a new class called FooServiceMock. You make it extend from AbstractMock<T>, where T is the class you are mocking.

As an example:

public class FooServiceMock extends AbstractMock<FooService> {

    public FooServiceMock() {
        super(FooService.class);
    }

It needs to have a constructor to call the super constructor passing the class being mocked too. Perhaps that could be optimised, I don't have too much time right now.

Next, you implement only those methods you expect to be called. For example:

public class FooServiceMock extends AbstractMock<FooService> {

    public FooServiceMock() {
        super(FooService.class);
    }

    /** 
     * this is a method which exists in FooService, 
     * but I want it to do something else.
     */
    public String sayHello(String name){
	    return "Hello " + name + 
              ", Foo here!  This is a stub method!";
    }

To use the mock, you'll notice that it doesn't extend the class which it mocks, which might be problematic... Well, there are good reasons. To do the mocking, the abstract base class is actually going to create a dynamic proxy which wraps itself behind the interface of the class being mocked. To the caller, it looks like the FooService, but it's not actually anything related to it. Anytime a call to the FooService is made, the first thing which the proxy does is log that call, using XStream to create an XML representation of the parameters being passed into the method. Then, the proxy goes and looks in the instance of the mock class to see if it can find the method being called (well at least a method which takes the same parameters and has the same name and return type). If it finds such a method, it calls it. In our example, the sayHello(String) method would get called. It returns the result if there is one, to the caller.

In the case where it cannot find the method, it throws an exception, because it assumes that if it was not implemented, you didn't expect it to be called. You could of course change this to suit your needs, maybe even calling the actual FooService.

So, how to you use the FooServiceMock to create a FooService instance which you can use to mock your service? In the test, where you setup the class under test, you do this:

    FooServiceMock fooService = new FooServiceMock();
	
    //perhaps tell it about objects you would
    //like it to return...
	
    instanceOfClassUnderTest.setFooService(
                               fooService.getMock());

The setFooService(FooService) method on the instance of the class you are testing is in my case present, but you might not have it and may need to use reflection to do it. It's a question of how testable you write your classes, and is a design choice.

The getMock() method on the AbstractMock class is the method which creates the dynamic proxy which wraps the instance of the mock.

You can now test the class. There is however still something useful you can do after testing, i.e. assert that the right calls were made in the correct order with the right parameters. You do this in the test class to:

    assertEquals(1, fooService.getCalls().size());
    assertEquals("[sayHello: <String>Ant</String>]", 
	                  fooService.getCalls().toString());

The above tests that the sayHello(String) method was called just once, and passed the name "Ant".

There are times when you might want to clear the call log, between parts of the test. For that, call the clearCalls() method on the mock object:

        fooService.clearCalls();

Finally, here is the code for the AbstractMock, which you might want to tweak, depending upon your needs:

/*  
 * Copyright (c) 2010 Ant Kutschera, maxant
 * 
 * This file is part of Ant Kutschera's blog, 
 * http://blog.maxant.co.uk
 * 
 * This is free software: you can redistribute
 * it and/or modify it under the terms of the
 * Lesser GNU General Public License as published by
 * the Free Software Foundation, either version 3 of
 * the License, or (at your option) any later version.
 * 
 * It is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied
 * warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
 * PURPOSE.  See the Lesser GNU General Public License for
 * more details. You should have received a copy of the
 * Lesser GNU General Public License along with the blog.
 * If not, see .
 */
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.ArrayList;
import java.util.List;
import com.thoughtworks.xstream.XStream;

/**
 * base class for mocks who need to log their calls.
 *
 * @author   Ant Kutschera
 */
public abstract class AbstractMock {

  /** 
   * xstream is used for creating a serialised xml
   * representation of objects
   */
  private XStream xstream = new XStream();

  /** every call to the class being mocked is logged */
  private List calls = new ArrayList();

  /** 
   * the interface which needs to be exposed, ie the
   * class of the object being mocked
   */
  private Class interfaceClass;

  /**
   * Creates a new AbstractMock object.
   *
   * @param  interfaceClass  the class of the object
   *                  being mocked
   */
  public AbstractMock(Class interfaceClass){
    this.interfaceClass = interfaceClass;
  }

  /**
   * @return  a list of strings, one for each call
   *    containing the method name followed by each
   *  parameter as an XML string as created by XStream.
   */
  public List getCalls() {
    return calls;
  }

  /**
   * resets the list of calls.
   */
  public void clearCalls() {
    calls.clear();
  }

  /**
   * @return  a proxy which wraps the instance, so
   *          that it can automatically handle
   *          unimplemented methods.
   */
  @SuppressWarnings({ "unchecked", "rawtypes" })
  public T getMock() {
    ClassLoader classLoader = getClass().getClassLoader();
    Class[] interfaces = new Class[] { interfaceClass };
    InvocationHandler ih = new InvocationHandler() {
	
      @Override
      public Object invoke(Object proxy, Method method, 
                          Object[] args) throws Throwable {

        Method m = getMethod(method);

        if (m == null) {
          //TODO you could change this if you like... 
          //maybe make it configurable?
          throw new RuntimeException("unexpected call to "+
                    "a method not in the mock: " + method);
        } else {
          String s = m.getName();
          if (args != null) {
            for (Object o : args) {
              s += ":" + xstream.toXML(o);
            }
          }

          //trim white space between xml tags as well as
          //new lines anywhere!
          s = s.replaceAll("\r", "").replace("\n", "");
          for (int i = 0; i < 10; i++) {
            char[] spaces = new char[i * 2];
            for (int j = 0; j < spaces.length; j++) {
              spaces[j] = ' ';
            }
            s = s.replaceAll(">" + new String(spaces) + 
                                                "<", "><");
          }

          //log the call...
          calls.add(s);

          //now call the mock implementation!
          return m.invoke(AbstractMock.this, args);
        }
      }
    };

    return (T) Proxy.newProxyInstance(classLoader, 
                                           interfaces, ih);
  }

  /**
   * the method with the same name/params, if it exists in
   * this class, otherwise null.
   */
  private Method getMethod(Method method) throws 
        IllegalAccessException, InvocationTargetException {

    for (Method domainObjectMethod : 
                                 getClass().getMethods()) {

      if (match(method, domainObjectMethod)) {
        return domainObjectMethod;
      }
    }

    return null;
  }

  private static boolean match(Method method1, 
                                          Method method2) {
      return namesAreEqual(method1, method2) 
	                    && typesAreEqual(method1, method2);
  }

  private static boolean namesAreEqual(Method method1, 
                                          Method method2) {
    return method2.getName().equals(method1.getName());
  }

  private static boolean typesAreEqual(Method method1, 
                                          Method method2) {
    Class[] types1 = method1.getParameterTypes();
    Class[] types2 = method2.getParameterTypes();

    if (types1.length != types2.length) {
      return false;
    }

    for (int i = 0; i < types1.length; ++i) {
      if (!types1[i].equals(types2[i])) {
        return false;
      }
    }
    return true;
  }
}

Social Bookmarks :

Taking Advantage of Parallelism

Sun, 16 May 2010 13:50:00 GMT

A while ago some colleagues attended a lecture where the presenter introduced the idea that applications may not take full advantage of the multi-core servers which are available today. The idea was that if you have two cores but a process which is running on a single thread, then all the work is done on one single core. Application servers help in this respect, because they handle multiple incoming requests simultaneously, by starting a new thread for each request. So if the server has two cores it can really handle two requests simultaneously, or if it has 6 cores, it can handle 6 requests simultanously.

So multi-core CPUs can help the performance of your server if you have multiple simultaneous requests, which is often the case when your server is running near its limit. But it's not often the case that you want your servers running close to the limit, so you typically scale out, by adding more nodes to your server cluster, which has a similar effect to adding cores to the CPU (you can continue to handle multiple requests simultaneously).

So once you have scaled up by adding more cores, and scaled out by adding more servers, how can you improve performance?

Some processes can be designed to be non-serial, especially in enterprise scenarios. The Wikipedia article on multi-core processors talks about this. Imagine a process which gathers data from multiple systems while preparing the data which it responds with. An example would be a pricing system. Imagine your local zoo offered a web service for purchasing individual tickets, say an entrance ticket, a voucher for some tasty elephant snacks for feeding time, a voucher for their shop for getting gifts before you leave, etc. Now imagine, you as an entrepreneur wanted to open a re-selling business offering packages for such products, so that you could offer a discount (greater sales volume, lower per-item profit).

A perfect way to do this would be to partner with the Zoo and offer a website which integrates with the zoo's web service and sells such packages. You implement your website, but then while doing performance tests, you notice that regardless of load, an average sales is taking 3 seconds to complete. You analyse the system and determine that its the zoo's web service which is the problem - each product quote is taking a full second to get. You talk to the zoo's IT department, and they confirm that this is known, but there is little they can do about it because in the back end they are using complex reservation systems which work out a dynamic market price which is based on supply and demand.

So, you go away and have a think, and remember that your data center has just just installed some servers with flashy new multi-core processors, which are costing you a fortune. Then it hits you! Getting a product quote is an independent task. So while getting quotes for the items in your package, you might as well do it in parallel, at the same time! Hang on now. You implemented your system in Java and the business logic is in EJBs, so there's no chance of spawning new threads, the container won't let you! Except... you implemented it on a shiny new Java EE 6 server! Hooray! Why? Because of the javax.ejb.Asynchronous annotation.

The Javadocs for this annotation state:

Used to mark a method as an asynchronous method or to designate all business methods of a class or interface as asynchronous.

So, what you do, is add this annotation to a business method of your EJB and the container runs it on a new thread. Brilliant! The only minor issue is how to get the result from your asynchronous calls, but that's relatively easy, using the Future<T> class. Consider the following method declaration:

/** * note that async methods implicitly start new transactions! * see ejb 3.1 spec, chapter 4.5.3. */ @Override @Asynchronous @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) public Future getOffer(Product product, UUID purchaseId) { . .
This EJB business method takes a product and purchase ID (effectively a session ID). It gets called by the container on a new thread, which implicitly means that it cannot re-use any existing transactions and must hence be run using a new transaction (see EJB specification, chapter 4.5.3). This method then calls the zoo's web service and when it gets a response, returns the response:

return new AsyncResult(o);
where "o" is an offer - the response from the web service call. The javax.ejb.AsyncResult<T> class is the standard implentation of a Future<T>, and simply wraps the actual object you want to return.

The EJB which started the async calls knows when they have all returned, because it can check the results isDone() method. As soon as the new thread completes, the container sets this method to return true.

So, the net effect is that instead of taking three seconds to get an offer for your package of three items, it only takes one second! A whopping 300% improvement!

The async annotation in the EJB spec is a welcome extension. If also offers the ability to cancel tasks as well as handle exceptions from the async calls (via ExecutionExceptions which wrap the problem). With it however, there are some dangers, as suggested in the Wikipedia article on multi-core processors. The fact that such async methods are forced to start a new transaction can cause problems. However, this is actually an advantage, because we are calling web services! Web services are inherently non-transactional. That means they are committed as soon as you get their response, not as soon as your transaction is committed. So in order to make web services XA (two phase commit) compatible, one normally adds a "commit" method to the web service which the caller must call when all their business logic is complete, or one adds a mechanism for cancelling such transactions.

I have used the new "inner" transaction which the container starts to track the state of these offers / sales. As soon as the web service returns, I write the result to a table in the database with its status "OFFERED", so that I know I have not actually purchased it. Regardless of errors in other calls to the web service, I am guaranteed to have such an entry for each successful web service result, due to the inner transaction. After I get the three offers, the orchestration EJB which called the async bean (which calls the web services), purchases the three offers. After each purchase, I update the status in the database to "BOUGHT", also using an inner transaction (REQUIRES_NEW), so that regardless of any failures in subsequent purchases, the state of individual purchases is known. Why go to this effort? Well, I have a business rule: my packages are sold as complete packages, or not at all! So if any single purchase fails, I need to cancel my other purchases. If I don't, I still owe the zoo for what I have purchased, and I didn't get any money from my customer, because the package failed!

The solution is to use the timer service from the EJB container. I schedule a method in an EJB to analyse my offers/purchases and for any incomplete session (purchase ID), I cancel all three offers and update my records to mark the purchases as "CANCELLED".

I've put this together as a sample app which includes EJB 3.1, Servlet 3.0, Inteceptors, JPA 2.0, the EJB Timer Service, Validation (JSR-303), JMS and JAX-WS, all based on the GlassFish 3 configuration which was blogged at GlassFish 3 In 30 Minutes. The source for this mega sample app can be downloaded here.

Some final quick notes:

1) A web service, and its client cannot be deployed in the same EAR! That makes sense in normal conditions, but for a sample app, where you want your server to call a test WS, it can cause trouble. So the zoo web service is deployed as a simple web project.

2) The Javadocs for the OrchestrationBean provide some more details of the business process.

© 2010 Ant Kutschera

Social Bookmarks :

GlassFish v3, JSF and Virtual Servers

Sat, 15 May 2010 14:26:00 GMT

Anyone wanting to run a JSF app on GlassFish may run into problems when migrating to production, if they are hosting multiple domains on their servers by using virtual hosts!

The default deployment, deploys your app to all virtual servers (except _asadmin). If you do that, you get lots of errors, and the app doesn't run.

The solution is easy, simply add the --virtualservers argument to the "deploy" command in the "asadmin" console, and supply the relevant virtual server (just the one).

Hopefully this will get fixed in GFv3.1!

Social Bookmarks :