Remote EJB stale connection? InvocationTargetException: Remoting connection channel has been closed

[sumitsu]

I'm encountering the following client-side exception when I try to invoke an AS7-deployed EJB 3 SLSB after a period of idle time:

java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at {client application}

        at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)

        at sun.rmi.transport.Transport$1.run(Transport.java:159)

        at java.security.AccessController.doPrivileged(Native Method)

        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)

        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)

        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

        at java.lang.Thread.run(Thread.java:619)

Caused by: java.lang.reflect.UndeclaredThrowableException

        at $Proxy4.doTask(Unknown Source)

        at {client application}

        ... 23 more

Caused by: java.io.IOException: Channel Channel ID abc6099f (outbound) of Remoting connection 00cb31d8 to null has been closed

        at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:110)

        at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:102)

        at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)

        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:517)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

        at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:382)

        at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:103)

        at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:55)

        ... 3 more

The client does not appear to automatically retry.  Here is my current jboss-ejb-client.properties file:

remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false

remote.connections=default

remote.connection.default.host={HOST}

remote.connection.default.port=19000

remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false

remote.connection.default.username={username}

remote.connection.default.password={password}

I don't recall having seen a similar "stale connection" effect when remotely invoking EJBs under previous version of JBoss.

• Have I incorrectly configured something, either for remoting on the server side, or on the client side?
• Is there anything I should do to make the AS7 EJB client handle this Exception internally, or do I just need to write client code to catch the InvocationTargetException and retry from there?

Thanks.

[jaikiran]

Which exact version of JBoss AS7? The attempt to reconnect to the broken connection will happen on the next invocation. I am not aware of any idle timeout on connections, although I have seen another user mentioning it when his client application was in a Resin server.

[sfcoy]

Network infrastructure can do horrible things to TCP connections that you think should be persistent.

If you're lucky, the offending piece of hardware will timeout the connection and close it properly. However, I've seen some devices that just just "forget" the connection so that both ends have no idea that they've been arbitrarily cut off. This has consequences for connection pools which can be exhausted by filling up with dead connections.

[sumitsu]

Thanks for the quick reply; I have noticed (as you said) that the following invocation attempt is successful, so if there's no way to configure the EJB client itself to retry, catching the InvocationTargetException and outside the client invocation might be a viable option.

I'm using JBoss AS 7.1.1-Final.

[jaikiran]

Stephen makes a good point about the disconnection.

However, the part I'm surprised about is why the reconnect attempt doesn't happen when you first invoke the method after some idle time. If the connection had disconnected sometime during the idle time, I would have expected JBoss Remoting to send a notification to the EJB client API about that disconnection so that it could enroll a reconnect handler such that it would try and reconnect when the next invocation happens. Apparently this isn't happening. It seems as if the notification is being sent only after a invocation is attempted and as a result, the reconnect can only be attempted in the next invocation. Is there an application you can share which reproduces this? And how long did you have to wait for this to be reproduced (few minutes)? In the meantime, I'll see if I can reproduce this locally.

Branden Smith wrote:

so if there's no way to configure the EJB client itself to retry, catching the InvocationTargetException and outside the client invocation might be a viable option.

No, don't do that. That's not right. The InvocationTargetException can happen for any number of reasons and trying to catch that exception and reinvoke a method isn't a good thing. We'll have to figure out what's wrong and fix it in the right place.

[sumitsu]

Thanks for looking into this; I'll see if I can pare down this application and post a zip of the important parts as a test case.  I'm not sure as to the exact period of time after which the connection appears to become stale; I'll see if I can find something in the logs which could speak to that.

[cclausen_]

Any progress or new information on this problem?

We are encountering it with our Swing-based client as well as with a JBoss7 that connects to another JBoss7. It does reconnect after the first Exception, but the first Exception is already too much considering that it leads to a visible error for the user and requieres the user to retry whatever he wanted to do.

We are using JBoss 7.1.1 Final

[chrkoelle]

I would also be interested if there a new information about this subject. We happen to see this behaviour also in our application from time to time. We use JBoss AS 7.1.2.Final.

[jaikiran]

See https://community.jboss.org/thread/205784