When running the mvn install for deployables\bindingcomponents\servicemix-http, you will get this error. The exception is due to an HTTPClient thread being locked out. The first request works fine but locks a thread, the next one fails as it tries to use the same thread out of the HTTPClient ThreadPool. The 3rd works but does the same thing. You eventually run out of threads (max 16) from the thread pool.
2008-07-22 17:41:34.09::INFO: jetty-6.1.6
2008-07-22 17:41:34.09::INFO: Started SelectChannelConnector@localhost:8192
<?xml version="1.0" encoding="UTF-8"?><jbi:message xmlns:jbi="http://java.sun.co
dl-first/types" name="Hello" type="msg:HelloRequest" version="1.0"> <jbi:part>
2008-07-22 17:41:34.179::WARN: EXCEPTION
2008-07-22 17:41:35.388::INFO: jetty-6.1.6
This is a known issue. The problem is due to Jetty HttpClient default threadpool size is 16 and maxConnectionPerAddress 32.
Workaround for this problem is to configure the threadpool size (maxConnectionsPerAddress*noOfHttpProviderEndpoints+1).
With latest release FUSE ESB 220.127.116.11 you can configure Jetty HttpClient threadpool size by adding
servicemix-http.jettyClientThreadPoolSize=<size you want>
in conf/component.properties file.
I compiled and debugged both the Jetty and SMX code and found out what the problem is. The problem is in the socket call. An http socket is created in Jetty's HTTPConnection for every invocation of the HttpProvider call. If the socket is idle and it has not exceeded the timeout limit, an incoming http request thread tries to use that instance of the ThreadPool. If the socket is stale, this creates an exception which renders the thread locked. You can't recover unless you reboot the server. As a result, the next request creates an instance of the ThreadPool which is bound by the limit of 16.
Eventually, this locks up the server. To work around the problem, I modified the jetty code to recreate the socket for every call. Is there a feature in Jetty's implementation of autohandling these types of errors? It would nice if we can get SMX to handle the issue seamlessly and recreate the socket if it is stale.
Can you log a Bug and submit testcase that reproduces the problem? It seems to be littlebit different than the problem I faced and suggested workaround for.
If I am not wrong there is a option in Jetty HttpClient to use non-blocking IO but currently it may have some connection issues which I heard are fixed in Jetty 7 and will be backported by Jetty team later in Jetty 6 as well. Currently FUSE ESB doesn't have option to switch between blocking and non-blocking IO.