Keep alive doesn't doesn't work for closed TCP/IP connection

[ropalka]

My usecase:

I'm using JBossWS Netty client (via JAX-WS proxy API) with keep-alive feature enabled (this is the default).
When remote http server goes down and up again all lately used cached connections on client side will fail with the following exception:

java.io.IOException: Could not transmit message">javax.xml.ws.WebServiceException: java.io.IOException: Could not transmit message
 at org.jboss.ws.core.jaxws.client.ClientImpl.handleRemoteException(ClientImpl.java:392)
 at org.jboss.ws.core.jaxws.client.ClientImpl.invoke(ClientImpl.java:302)
 at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:170)
 at org.jboss.ws.core.jaxws.client.ClientProxy.invoke(ClientProxy.java:150)
 at $Proxy13.echo(Unknown Source)
 at org.jboss.test.ws.jaxws.endpoint.jse.UsecasesTestCase.invokeEndpoint1(UsecasesTestCase.java:128)
 at org.jboss.test.ws.jaxws.endpoint.jse.UsecasesTestCase.testTwoPortsAndLongPaths(UsecasesTestCase.java:86)
Caused by: java.io.IOException: Could not transmit message
 at org.jboss.ws.core.client.transport.NettyClient.invoke(NettyClient.java:194)
 at org.jboss.ws.core.client.HTTPRemotingConnection.invoke(HTTPRemotingConnection.java:150)
 at org.jboss.ws.core.client.SOAPProtocolConnectionHTTP.invoke(SOAPProtocolConnectionHTTP.java:69)
 at org.jboss.ws.core.CommonClient.invoke(CommonClient.java:344)
 at org.jboss.ws.core.jaxws.client.ClientImpl.invoke(ClientImpl.java:290)
 ... 31 more
Caused by: java.util.concurrent.ExecutionException: java.nio.channels.ClosedChannelException
 at org.jboss.ws.core.client.transport.WSResponseHandler$FutureResult.get(WSResponseHandler.java:178)
 at org.jboss.ws.core.client.transport.WSResponseHandler$FutureResult.get(WSResponseHandler.java:113)
 at org.jboss.ws.core.client.transport.NettyClient.invoke(NettyClient.java:156)
 ... 35 more
Caused by: java.nio.channels.ClosedChannelException
 at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:624)
 at org.jboss.netty.channel.socket.nio.NioWorker.write(NioWorker.java:353)
 at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:124)
 at org.jboss.netty.channel.Channels.write(Channels.java:889)
 at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:76)
 at org.jboss.netty.channel.Channels.write(Channels.java:869)
 at org.jboss.netty.channel.Channels.write(Channels.java:818)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:236)
 at org.jboss.ws.core.client.transport.NettyTransportOutputStream.close(NettyTransportOutputStream.java:142)
 at org.jboss.ws.core.client.transport.NettyClient.writeRequest(NettyClient.java:240)
 at org.jboss.ws.core.client.transport.NettyClient.invoke(NettyClient.java:147)
 ... 35 more

Our client side Netty connections cache have to implement algorithm that will detect, if cached connection is still alive (the one going to be returned to client).

[ropalka]

Steps to reproduce the issue:

svn checkout the following branches:
* spi/branches/ropalka
* common/branches/ropalka
* framework/branches/ropalka
* container/jboss50/branches/ropalka
* stack/native/branches/ropalka
Do 'mvn clean install' for:
* spi/branches/ropalka
* common/branches/ropalka
* framework/branches/ropalka
* container/jboss50/branches/ropalka
Comment out the following line in org.jboss.test.ws.jaxws.endpoint.jse.UsecasesTestCase.setUp():

System.setProperty(Constants.HTTP_KEEP_ALIVE, Boolean.FALSE.toString());

Run the test:

mvn -Ptestsuite,hudson,jboss510 -Dtest=**/jaxws/endpoint/jse/UsecasesTestCase test

[ropalka]

Cross reference issue: JBWS-2753

[asoldano]

Fixed in ropalka branch. Richard, please merge along with all your stuff once you're done.

[ropalka]

The fix is not acceptable :(
We cannot simply catch exception and resend the message.
This fix won't work for long streaming attachments :( It just works for really simple usecases.
The correct fix will be to detect closed connections before writing to the stream.

[asoldano]

Richard, as discussed on IRC, I don't see the problem you're talking about, please provide a testcase demonstrating your claims.

The fix is not acceptable :(
We cannot simply catch exception and resend the message.
This fix won't work for long streaming attachments :( It just works for really simple usecases.
The correct fix will be to detect closed connections before writing to the stream.

[ropalka]

Richard, as discussed on IRC, I don't see the problem you're talking about, please provide a testcase demonstrating your claims.

I finally have attachments working in embedded Netty HTTP server.
I provided the test case as I promissed, just uncomment the System.setProperty() again to reproduce the issue.

There's another how-to-fix that comes in my mind :) We could flush HTTP headers before writing HTTP body to the stream. This should be enough to make your fix (catching the exception and resending the message) working.

[ropalka]

IOW your fix is acceptable and almost correct (i.e. it is acceptable to catch the exception and resend). We just need to get Connection closed exception before reading from input streams representing attachments. This could be achieved by flushing HTTP headers before writing HTTP body as suggested above.

[asoldano]

Richard, the testcase you provided is not valid as it goes against the DataSource API, see http://java.sun.com/javaee/5/docs/api/javax/activation/DataSource.html#getInputStream(). Basically each invocation of DataSouce#getInputStream MUST return a new InputStream object and the stream must be positioned at the beginning of the data. That's the reason of the failure.

[ropalka]

You're right. I should start reading javadocs finally ;)