<< January 2010 | Home | March 2010 >>

Transfer-Encoding: chunked

The J2ME HTTPConnection which comes with MIDP lets you make HTTP requests to your server. It doesn't do much at a high level, for example the API doesn't have methods like addCookie() - you need to manually add them with a request header. But the implementation is clever enough to turn any request body which is greater than around 2Kb into a chunked request.

With HTTP 1.0, the request had to contain a header called Content-Length which told the server how many bytes to read off the input stream when reading the body. HTTP 1.1 introduced the Transfer-Encoding header, which lets the client omit the Content-Length header, and instead create chunks of request body, which optimises the upload meaning that a) the server can start processing before it has everything, and b) more importantly for J2ME where memory might be a valuable resource, it lets the client send a bit of the request, free up that allocated memory and then send some more of the request.

For a POST request, with no chunking, the headers and body might look like this:

	POST /log.jsp HTTP/1.1
	User-Agent: Mozilla/4.0 (maxant J2ME Client) 
	Accept: text/html,application/xhtml+xml,application/xml
	Content-Type: application/x-www-form-urlencoded
	Content-Length: 51
	Host: wwwchaseamatecom:8089

	problem=Failed%20to%20get%20installation%20response


Chunked, that becomes:

	POST /ota/addInstallation.jsp HTTP/1.1
	User-Agent: Mozilla/4.0 (maxant J2ME Client) 
	Accept: text/html,application/xhtml+xml,application/xml
	Content-Type: application/x-www-form-urlencoded
	Host: wwwchaseamatecom:8089
	Transfer-Encoding: chunked

	problem=Failed%20to%20get%20installation%20response


You'll notice that the body of the second example, "problem=..." doesn't contain chunk headers (search Wikipedia for chunking to see an example). The reason is that I copied that text out of the TCP/IP Monitor in Eclipse 3.5 and it seems kind enough to hide that information and simply show you the unchunked body. Not great if you aren't expecting that, somewhat useful none the less.

Anyway, about time I got to the point of this blog article. Any version of Tomcat before 5.5.28 or 6.0.21 (ie. the latest versions at the time of writing!) had a nasty bug in them whereby POST requests did NOT get the body parsed and put into request.getParameter(String) or request.getQueryString(). Very nasty indeed for J2ME developers. I trawled the internet and found people were having problems from around 2002 when HTTP 1.1 became available in J2ME. Only with the following bug fix has this finally been corrected on Tomcat, some 8 years later.

    https://issues.apache.org/bugzilla/show_bug.cgi?id=37794

Work arounds included using Apache to put together the chunks before sending the request on to Tomcat, or manually splitting the request on the device, into small chunks and upload each one and then joining them together at the end - what a load of work to have to do!

I'm working with an old 5.5.9 Tomcat installation and have just upgraded it to 5.5.28 to fix this problem.

However before I found this bug report I did consider creating a filter and configuring it in my web.xml. The idea was that the filter would read the request body, URL decode it, unlock the requests parameter map (Catalina has a lockable map which is locked after parsing the request), and put the request bodies request parameters into the map. Something like this:

 


/*  
 * Copyright (c) 2010 Ant Kutschera, maxant
 * 
 * This file is part of Ant Kutschera's blog, http://blog.maxant.co.uk
 * 
 * This is free software: you can redistribute it and/or modify
 * it under the terms of the Lesser GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * It is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * Lesser GNU General Public License for more details.
 * You should have received a copy of the Lesser GNU General Public License
 * along with Foobar.  If not, see .
 */
import java.io.IOException;
import java.io.InputStream;
import java.net.URLDecoder;
import java.util.HashMap;
import java.util.Map;
import java.util.Properties;
import java.util.Map.Entry;

import javax.servlet.http.HttpServletRequest;

//nasty, but we are afterall fixing a bug in catalina!
import org.apache.catalina.util.ParameterMap;

public final class RequestHandler implements Filter {

	private static final String TRANSFER_ENCODING = "Transfer-Encoding";
	private static final String CHUNKED = "chunked";

	public void destroy() {
	}

	public void init(FilterConfig filterConfig) throws ServletException {
	}
	
	public void doFilter(ServletRequest request, ServletResponse response,
			FilterChain chain) throws IOException, ServletException {

		String header = request.getHeader(TRANSFER_ENCODING);
		
		InputStream is = null;
		try{
			if(header != null && header.equals(CHUNKED)){
				//time to work our magic

				((ParameterMap)request.getParameterMap()).setLocked(false);
				is = request.getInputStream();

				//read the body as a map
				Properties props = new Properties();
				props.load(is);
				
				//get the encoding to use for url-decoding
				String enc = request.getCharacterEncoding();
				if(enc == null){
					enc = "UTF-8";
				}

				for(Entry e : props.entrySet()){
					
					String key = (String) e.getKey();

					//need to decode it!
					key = URLDecoder.decode(key, enc);

					if(!request.getParameterMap().containsKey(key)){
						String val = (String) e.getValue();

						//need to decode it!
						val = URLDecoder.decode(val, enc);

						request.getParameterMap().put(key, val);
					}
				}
			}				
				
			((ParameterMap)request.getParameterMap()).setLocked(true);
				
		}finally{
			if(is != null) is.close();
		}
	}
}


I wasted hours because of this rubbish bug. The problem is that its full of red herrings, making you think for example that because the monitor shows the request body unchunked, that the device isn't chunking the body properly. The web is also full of very old posts related to servers not being HTTP 1.1 compatible. And there are lots of work arounds which people went to the effort of because they had no other choice until most recently. So hopefully if you have had chunking problems with J2ME, you found this article without too much hassle!

Copyright (c) 2010 Ant Kutschera
 

 

Tags : , ,
Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

A J2ME Library and a simple HTTP Service Framework

J2ME's support for calling a server is rather simple and low level. Not only do you have to deal with the HTTP Connection at a low level, there is no high level support for cookies, authentication or remote procedure calling. So if you want to pass an object to the server and get a response, you need to figure out how to do that. XML over HTTP is one solution, but presents its own problems like the serialisation and deserialisation of objects, not to mention higher network traffic because of the meta-data held within the XML. JAX Binding is certainly not supported in J2ME-land which results in you having to use a SAX parser.

In previous projects I have toyed with a simple way of providing services over JSPs, which take and receive delimited text. The idea is to implement your own simple serialization and deserialisation of simple objects allowing you to make simple calls to the server and receive simple responses. I purposefully used the word "simple" four times in that last sentence to impress upon you the idea that server calls should be kept simple.

Take for example a J2ME application which tracks a GPS location. To send the location of the user it can simply send a line of text like this:

    006.574438|045.453345|11022344843373

What's it mean?

    longitude | latitude | timestamp

The serialising and deserialising of the data is VERY simple using a StringTokenizer (erm, which doesn't exist in J2ME, so see later!). And the server could respond with a simply OK or it might want to take the opportunity to update the client with some traffic information:

 M4|4|6|20|Accident

which means:

    road | from junction | to junction | how many minutes delay | reason

Most server calls really can be that simple, especially when being involved in J2ME applications which tend to be relatively simple themselves.

So the above presents a few questions... How secure is the data and how do you know who the update is coming from? Well the data should be sent over SSL to ensure that its secure and if the data is sent over an authenticated session, then the server knows who the logged in user is. But to get a log in and session to work, you need two things, namely cookies (for the session ID to be passed between client and server) and some form of authentication. Cookies in J2ME aren't too easy to handle since there is no built in API for handling them at a high level. You can set a cookie in the request header, but storing cookies from responses is the hard part. I implemented rudimentary cookie handling by sending the response to a method which checks for any set-cookie headers and adds them to the cache as well as cleaning out expired entries. When a request is sent, I call a method which adds all relevant cookies to the request header. I have not implemented the RfC for cookies and don't handle the differences between Cookie and Cookie2. In fact I didn't even go as far as checking the path of the cookie before sending it to the server, because in my environment, it's not even relevant. A proper cookie implementation would need to do those things and more, and perhaps one day, such a library will exist.. I did manage to find this which refers to the JCookie sourceforge project and J2ME, but checking out its site I couldn't find anything that would work with J2ME.

HTTP authentication I originally handled by adding the "authorization" request header and applying a Base64 encoded basic authentication string to it. This caused its own problems, because J2ME doesn't have a Base64 encoder out of the box. Luckily the org.apache.common.codecs.binary.Base64 class works (which I took from the Apache Codecs 1.3 library. It depends upon the following classes which are in the distributable JAR and also J2ME compatible: BinaryDecoder, BinaryEncoder, Decoder, DecoderException, Encoder, EncoderException, all found in the org.apache.commons.codec package.

I ran into a different problem when I wanted my web application for the web site to be the same as the web application for my services. Namely, for services I wanted to use basic authentication and for the web site I wanted to use form login. The servlet spec doesn't let you define more than one login-config per web-app! So the J2ME class I had written for accessing services (ServerCaller) had to be extended, but that wasn't hard. When a web application using Form Login needs you to authenticate, it simply returns the login form instead of the expected response. If you parse the response and check to see if it is your login form, and then simply submit that form with your username and password, the server then logs you in and returns a 302 code telling you to call the original page again. Assuming you provided the correct login credentials, it all works. So my class recursed into itself if it spotted the login form and that was enough to make it work transparently.

The next problem was the parsing of the responses in order to deserialise them. For this I created a very simple StringTokenizer, since neither CLDC nor MIDP gives you one :-( The implementation is below.

URL encoding and decoding is also important because request parameters (for a GET its the request string, for a POST the request body). Luckily there are some options out there. Catalina has one in its util package, which almost works for J2ME. So I did a quick fix to it to remove the dependency on java.util.BitSet which also doesn't exist in J2ME.

The last problem I had related to chunking of the request by the HTTPConnection, which you can read more about here.

So, finally to some code! The magic class is the ServerCaller which you can download below. It is abstract and to use it you simply need to extend it, for example:


package uk.co.maxant.cam.c;

import java.util.Vector;

import uk.co.maxant.cam.m.MasterData;
import uk.co.maxant.j2me.core.CredentialsProvider;
import uk.co.maxant.j2me.core.ServerCaller;
import uk.co.maxant.j2me.core.StringTokenizer;

/**
 * calls the server to get master data like roles and the version of the latest available download.
 */
public abstract class MasterDataGetter extends ServerCaller implements ServerCaller.Listener {

	private ExceptionHandler eh;
	
	public MasterDataGetter(CredentialsProvider credentialsProvider, ExceptionHandler eh) {
		super(credentialsProvider, Constants.getBaseUrl(), "ota/getMasterData.jsp");
		this.eh = eh;
		setListener(this);
	}
	
	public void callServer(){
		properties.clear();
		properties.put("installUid", String.valueOf(InstallHelper.getInstallationUid(eh)));

		doCallAsync();
	}
	
	protected void handleSuccess(String str) {
		StringTokenizer lines = new StringTokenizer(str, "|");
		String version = lines.nextToken().trim();
		String sessionId = lines.nextToken().trim();
		String screenName = lines.nextToken().trim();
		String roles = lines.nextToken().trim();
		StringTokenizer rs = new StringTokenizer(roles, ",");
		Vector rolesv = new Vector();
		while(rs.hasMoreTokens()){
			rolesv.addElement(rs.nextToken());
		}
		
		String s = lines.nextToken().trim();
		boolean isSendDebugLog = s.equalsIgnoreCase("true");
		
		MasterData masterData = new MasterData(version, rolesv, sessionId, screenName, isSendDebugLog);
		onSuccess(masterData );
	}
	
}


An example of a JSP for the server is:


<%@page import="uk.co.maxant.cam.web.VersionHelper"%>
<%@page import="org.apache.log4j.Logger"%>
<%@page import="java.io.FileInputStream"%>
<%@page import="java.util.Properties"%>
<%@page import="java.io.RandomAccessFile"%>
<%@page import="java.io.File"%>
<%@page import="uk.co.maxant.util.ServiceLocator"%>
<%@page import="uk.co.maxant.cam.services.OrchestrationService"%>
<%@page import="uk.co.maxant.cam.data.Party"%>
<%@page import="java.util.List"%>
<%@page import="uk.co.maxant.cam.services.OrchestrationService.MasterData"%>


<%
Logger log = Logger.getLogger(this.getClass().getCanonicalName());
String user = "";
String roles = "";

//either they are logged in, or we can check their login against the authentication header, since its ALWAYS sent
try{
	//we know who they are. if the installUid is known, lets add it to their profile!
	String installUid = request.getParameter("installUid");
	if(installUid != null && installUid.equals("null")) installUid = null;

	String auth = request.getHeader("authorization");

	OrchestrationService os = ServiceLocator.getInstance().getOrchestrationService();
	MasterData md = os.getMasterData(auth, installUid);
	if(md.party != null) user = md.party.getScreenname();
	for(String role : md.roles){
		roles += role + ",";
	}
	String version = VersionHelper.getVersionFromJad(request);
	
	boolean sendDebugLog = true; //change to false if we get swamped. but really, we shouldnt ever get this stuff.
	
	%>OK
	<%=version%>|
	<%=session.getId()%>|
	<%=user%>|
	<%=roles%>|
	<%=sendDebugLog%>|
	<%
}catch(Exception e){
	log.warn("failed to read master data", e);
	%>ERROR
	<%=e.getMessage() %>
	<%
}
%>




The result, is the maxant J2ME library, which runs on CLDC 1.1 and MIDP 2.0, and can be downloaded here. It includes classes for all of the following, and more, and is released under the LGPL Licence:

  • URLEncoder - from Apache Catalina
  • Base64 - from Apache Commons Codec
  • CredentialsProvider - an interface which the ServerCaller uses to get server credentials for logging in
  • Formatter - formats for example decimals nicely
  • GAUploader - loads an event to Google Analytics
  • Math - some math stuff thats not in CLDC
  • Point - a 2 dimensional point in a plane
  • ServerCaller - a high level API for calling a server, as described above
  • StringTokenizer - simple, and a lot like that from J2SE
  • URL - an encapsulation of a protocol, domain, port and page
  • ImageHelper - code taken from the web for scaling images
  • DirectoryChooser - a UI for choosing a local directory on the device
  • FileChooser - a UI for choosing a local file on the device


To use the library, add it to the "lib" folder of your J2ME project, then ensure its on the build path, and in Eclipse Pulsar, you need to also check the box in the build path on the last tab, to ensure the classes in the JAR are exported to your JADs JAR.

Copyright (c) 2010 Ant Kutschera

Tags : , ,
Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Base X Encoding

Ever needed to shorten a number so that its easier to remember? Or provide someone with a temporary PIN which is short enough to remember, but long enough to pretty much ensure it wont be randomly guessed by someone else? Converting a binary number into a hexadecimal is exactly the process used in such cases. But hexadecimal only has 16 characters in its "dictionary". Base64 is the next step up, with a bigger dictionary containing all alphanumerics (upper and lower case) as well as "/" and "+".

I need a solution which didn't contain certain characters. For example, its easy to mix up an O with a 0. Or an I,l and a 1. I wanted a solution whereby I could encode a number, but using my own definition of the dictionary. So I built just such a solution. You can see the source code below. It contains a main method which runs a simple test, the output of which is:

    Original: 123456789012345678901234567890
    encoded: 2aYls9bkamJJSwhr0
    decoded: 123456789012345678901234567890
    Passed! decoded value is the same as the original.

As you can see, the encoded version is only half as long as the input. Using an 89 character dictionary, it gets even shorter:

    encoded: "9Kgbz])M.w8KgK

The implementation uses the BigInteger class from Java, so you can encode REALLY big numbers. My phone number is now only 5 characters long and really easy to remember:

    rDm3T


/*  
 * Copyright (c) 2010 Ant Kutschera, maxant
 * 
 * The code below is free software: you can redistribute it and/or modify
 * it under the terms of the Lesser GNU General Public License as published by
 * the Free Software Foundation, either version 3 of the License, or
 * (at your option) any later version.
 * 
 * The code in this file is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * Lesser GNU General Public License for more details.
 * You should have received a copy of the Lesser GNU General Public License
 * along with Foobar.  If not, see http://www.gnu.org/licenses/.
 */

package uk.co.maxant.util;

import java.math.BigInteger;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

/**
 * allows you to convert a whole number into a compacted representation of that number,
 * based upon the dictionary you provide. very similar to base64 encoding, or indeed hex
 * encoding.
 */
public class BaseX {

	/**
	 * contains hexadecimals 0-F only.
	 */
	public static final char[] DICTIONARY_16 = 
		new char[]{'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F'};

	/**
	 * contains only alphanumerics, in capitals and excludes letters/numbers which can be confused,
	 * eg. 0 and O or L and I and 1.
	 */
	public static final char[] DICTIONARY_32 = 
		new char[]{'1','2','3','4','5','6','7','8','9','A','B','C','D','E','F','G','H','J','K','M','N','P','Q','R','S','T','U','V','W','X','Y','Z'};

	/**
	 * contains only alphanumerics, including both capitals and smalls.
	 */
	public static final char[] DICTIONARY_62 = 
		new char[]{'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z'};

	/**
	 * contains alphanumerics, including both capitals and smalls, and the following special chars:
	 * +"@*#%&/|()=?'~[!]{}-_:.,; (you might not be able to read all those using a browser!
	 */
	public static final char[] DICTIONARY_89 = 
		new char[]{'0','1','2','3','4','5','6','7','8','9','A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','a','b','c','d','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z','+','"','@','*','#','%','&','/','|','(',')','=','?','~','[',']','{','}','$','-','_','.',':',',',';','<','>'};
	
	protected char[] dictionary;

	/**
	 * create an encoder with the given dictionary.
	 * 
	 * @param dictionary the dictionary to use when encoding and decoding.
	 */
	public BaseX(char[] dictionary){
		this.dictionary = dictionary;
	}
	
	/**
	 * creates an encoder with the {@link #DICTIONARY_62} dictionary.
	 *  
	 * @param dictionary the dictionary to use when encoding and decoding.
	 */
	public BaseX(){
		this.dictionary = DICTIONARY_62;
	}
	
	/**
	 * tester method.
	 */
	public static void main(String[] args) {
		String original = "123456789012345678901234567890";
		System.out.println("Original: " + original);
		BaseX bx = new BaseX(DICTIONARY_62);
		String encoded = bx.encode(new BigInteger(original));
		System.out.println("encoded: " + encoded);
		BigInteger decoded = bx.decode(encoded);
		System.out.println("decoded: " + decoded);
		if(original.equals(decoded.toString())){
			System.out.println("Passed! decoded value is the same as the original.");
		}else{
			System.err.println("FAILED! decoded value is NOT the same as the original!!");
		}
	}

	/**
	 * encodes the given string into the base of the dictionary provided in the constructor.
	 * @param value the number to encode.
	 * @return the encoded string.
	 */
	public String encode(BigInteger value) {
		
		List<Character> result = new ArrayList<Character>();
		BigInteger base = new BigInteger("" + dictionary.length);
		int exponent = 1;
		BigInteger remaining = value;
		while(true){
			BigInteger a = base.pow(exponent); //16^1 = 16
			BigInteger b = remaining.mod(a); //119 % 16 = 7 | 112 % 256 = 112
			BigInteger c = base.pow(exponent - 1);
			BigInteger d = b.divide(c);
			
			//if d > dictionary.length, we have a problem. but BigInteger doesnt have
			//a greater than method :-(  hope for the best. theoretically, d is always 
			//an index of the dictionary!
			result.add(dictionary[d.intValue()]);
			remaining = remaining.subtract(b); //119 - 7 = 112 | 112 - 112 = 0

			//finished?
			if(remaining.equals(BigInteger.ZERO)){
				break;
			}

			exponent++;
		}

		//need to reverse it, since the start of the list contains the least significant values
		StringBuffer sb = new StringBuffer();
		for(int i = result.size()-1; i >= 0; i--){
			sb.append(result.get(i));
		}
		return sb.toString();
	}
	
	/**
	 * decodes the given string from the base of the dictionary provided in the constructor.
	 * @param str the string to decode.
	 * @return the decoded number.
	 */
	public BigInteger decode(String str) {

		//reverse it, coz its already reversed!
		char[] chars = new char[str.length()];
		str.getChars(0, str.length(), chars, 0);

		char[] chars2 = new char[str.length()];
		int i = chars2.length -1;
		for(char c : chars){
			chars2[i--] = c;
		}

		//for efficiency, make a map
		Map<Character, BigInteger> dictMap = new HashMap<Character, BigInteger>();
		int j = 0;
		for(char c : dictionary){
			dictMap.put(c, new BigInteger("" + j++));
		}

		BigInteger bi = BigInteger.ZERO;
		BigInteger base = new BigInteger("" + dictionary.length);
		int exponent = 0;
		for(char c : chars2){
			BigInteger a = dictMap.get(c);
			BigInteger b = base.pow(exponent).multiply(a);
			bi = bi.add(new BigInteger("" + b));
			exponent++;
		}

		return bi;
		
	}
	
}


   
Copyright (c)2010 Dr Ant Kutschera

Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!

Generating animated GIFs for AJAX loading

Cool! http://www.ajaxload.info/

Tags :
Social Bookmarks :  Add this post to Slashdot    Add this post to Digg    Add this post to Reddit    Add this post to Delicious    Add this post to Stumble it    Add this post to Google    Add this post to Technorati    Add this post to Bloglines    Add this post to Facebook    Add this post to Furl    Add this post to Windows Live    Add this post to Yahoo!