HTTP Provider BoundedThreadPool problem

[rresma]

There is an issue with the use of org.mortbay.jetty.client.HttpClient in HttpProviderEndpoint.  It seems that the NMR is throwing a BoundedThreadPool exception which is caught in HttpProviderEndpoint.Exchange:

protected void onException(Throwable throwable).

From debugging, it looks like it is a jetty issue as a previously used HttpClient being reused.  SMX may not be releasing the connection properly.  To work around the problem a new instance of org.mortbay.jetty.client.HttpClient is being created, started, and stopped for every request which is inefficient.

Regards,

Ramon

[rresma]

When running the mvn install for deployables\bindingcomponents\servicemix-http, you will get this error.  The exception is due to an HTTPClient thread being locked out.  The first request works fine but locks a thread, the next one fails as it tries to use the same thread out of the HTTPClient ThreadPool.  The 3rd works but does the same thing.  You eventually run out of threads (max 16) from the thread pool.

2008-07-22 17:41:34.09::INFO:  jetty-6.1.6

2008-07-22 17:41:34.09::INFO:  Started SelectChannelConnector@localhost:8192

<?xml version="1.0" encoding="UTF-8"?><jbi:message xmlns:jbi="http://java.sun.co

m/xml/ns/jbi/wsdl-11-wrapper" xmlns:msg="http://servicemix.apache.org/samples/ws

dl-first/types" name="Hello" type="msg:HelloRequest" version="1.0">  <jbi:part>

   <msg:GetPerson><msg:personId>id</msg:personId></msg:GetPerson>  </jbi:part></

jbi:message>

2008-07-22 17:41:34.179::WARN:  EXCEPTION

java.io.InterruptedIOException

        at org.mortbay.jetty.client.HttpConnection.handle(HttpConnection.java:17

1)

        at org.mortbay.jetty.client.HttpClient$SocketConnector$1.run(HttpClient.

java:321)

        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool

.java:450)

2008-07-22 17:41:35.388::INFO:  jetty-6.1.6

[ubhole]

This is a known issue. The problem is due to Jetty HttpClient default threadpool size is 16 and maxConnectionPerAddress 32.

Workaround for this problem is to configure the threadpool size (maxConnectionsPerAddress*noOfHttpProviderEndpoints+1).

With latest release FUSE ESB 3.3.1.4 you can configure Jetty HttpClient threadpool size by adding

servicemix-http.jettyClientThreadPoolSize=&lt;size you want&gt;

in conf/component.properties file.

Regards,

Ulhas Bhole

[rresma]

I compiled and debugged both the Jetty and SMX code and found out what the problem is.  The problem is in the socket call.  An http socket is created in Jetty's HTTPConnection for every invocation of the HttpProvider call.  If the socket is idle and it has not exceeded the timeout limit, an incoming http request thread tries to use that instance of the ThreadPool.  If the socket is stale, this creates an exception which renders the thread locked.  You can't recover unless you reboot the server.  As a result, the next request creates an instance of the ThreadPool which is bound by the limit of 16.

Eventually, this locks up the server.  To work around the problem, I modified the jetty code to recreate the socket for every call.  Is there a feature in Jetty's implementation of autohandling these types of errors?  It would nice if we can get SMX to handle the issue seamlessly and recreate the socket if it is stale.

[ubhole]

Can you log a Bug and submit testcase that reproduces the problem? It seems to be littlebit different than the problem I faced and suggested workaround for.

If I am not wrong there is a option in Jetty HttpClient to use non-blocking IO but currently it may have some connection issues which I heard are fixed in Jetty 7 and will be backported by Jetty team later in Jetty 6 as well. Currently FUSE ESB doesn't have option to switch between blocking and non-blocking IO.

--Ulhas