Leaks in pinger when using manual reconnection?

[jmesnil]

== Setup ==

* 1 single server (no backup)
* JMS Connection Factory configured with:

 <retry-interval>1000</retry-interval>
 <retry-interval-multiplier>1.0</retry-interval-multiplier>
 <reconnect-attempts>-1</reconnect-attempts>
 <failover-on-server-shutdown>true</failover-on-server-shutdown>
 <call-timeout>5000</call-timeout>

With this configuration, clients will try to reconnect indefinitely on the single server.

* One JMS ExceptionListener for manual reconnection:

private ExceptionListener exceptionListener = new ExceptionListener()
{
 public void onException(JMSException arg0)
 {
 System.out.println("TopicExample.onException()");
 disconnect();
 reconnect();
 }
};

disconnect() closes the current JMS connection (if any)
reconnect() recreates all the JMS resources from JNDI

* The client registers a MessageListener (to print received messages) and send messages every 0.5s

I've added traces in the Pinger handlePacket() method to display when pings where received for which remoting connections.

== Steps ==

1. start the server

2. start the client

=> On the client, we create 2 remoting connections (1 when calling JMS cf.createConnection(), the other when calling JMS conn.createSession())
=> We receive pings for these 2 remoting connections

3. kills the server
=> On the client, the connection failure is detected:

 6 aout 2009 13:29:57 org.jboss.messaging.core.logging.Logger warn
 ATTENTION: Connection failure has been detected The connection was closed by the server:5
 6 aout 2009 13:29:57 org.jboss.messaging.core.logging.Logger warn
 ATTENTION: Connection failure has been detected The connection was closed by the server:5

4. Restart the server
=> On the client, we try to reattach the sessions to the servers. This code creates new RemotingConnections *and new Pingers*

 6 ao?t 2009 13:30:28 org.jboss.messaging.core.logging.Logger warn
 ATTENTION: 9342944 Session not found on server when attempting to re-attach
 6 ao?t 2009 13:30:28 org.jboss.messaging.core.logging.Logger warn
 ATTENTION: 11968595 Session not found on server when attempting to re-attach

5. Client's JMS ExceptionListener is called
=> when we try to close the current connection, it fails:

 javax.jms.JMSException: Timed out waiting for response when sending packet 74
 at org.jboss.messaging.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:328)
 at org.jboss.messaging.core.client.impl.ClientConsumerImpl.doCleanUp(ClientConsumerImpl.java:753)
 at org.jboss.messaging.core.client.impl.ClientConsumerImpl.close(ClientConsumerImpl.java:318)
 at org.jboss.messaging.jms.client.JBossMessageConsumer.close(JBossMessageConsumer.java:153)
 at org.jboss.messaging.jms.client.JBossSession.close(JBossSession.java:275)
 at org.jboss.messaging.jms.client.JBossConnection.close(JBossConnection.java:259)
 at org.jboss.jms.example.TopicExample.disconnect(TopicExample.java:112)
 at org.jboss.jms.example.TopicExample$2.onException(TopicExample.java:69)
 at org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener.connectionFailed(JBossConnection.java:545)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:402)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:244)
 at org.jboss.messaging.core.client.impl.ConnectionManagerImpl$Channel0Handler$1.run(ConnectionManagerImpl.java:1119)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
 at java.lang.Thread.run(Thread.java:613)
 Caused by: MessagingException[errorCode=3 message=Timed out waiting for response when sending packet 74]
 ... 15 more

5b. Still in the JMS ExceptionListener, we reconnect to the Server starting from JNDI resources. This creates new RemotingConnections *and new Pingers*

6. The client can once again produce and consume messages

*but* the client receives pings for 4 connections!
It receives pings for the 2 remoting connections which were created when reconnecting.
But it also receives pings for the 2 remoting connections which were created when reattaching the session at step #4.
The exception at step #5 prevented the JMS connection to be closed properly, including closing its associated remoting connection(s) and its pinger(s)

If I kill/restart the server once more, I have now 6 pingers on the client.

[timfox]

I've found a similar issue:

If I create a connection factory, then create some connections with it, so there are several underlying remoting connections in the connection manager.

Then I manually fail one of the connections. This causes the following code to get executed in connection manager:

else
 {
 log.info("Not attempting failover");

 // Just fail the connections

 closePingers();

 failConnection(me);
 }

As you can see all the pingers get closed, but the actual connections remain in the connection manager.

Now if you wait longer than connection ttl, the server will close the connection from the server side since it didn't receive any ping from the connection which is still open.

[timfox]

Jeff, can you try again with TRUNK? I made some fixes for possibly related issues.

[sv_srinivaas]

Hi, I’m facing the exact same issue what is rerpoted in HORNETQ-2. I tried the jca-config example that comes with hornetq. Things worked fine as long as the network was connected and the moment I disconnected the server from the network and reconnected it, I got the below error in my server and after the mdb stopped consuming messages

[exec] WARNING: Connection failure has been detected: Did not receive ping from /172.18.xxx.yyy:2100. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]

Pls let me know how to fix this issue. Thanks!!!

Environment Details:
Windows Xp (Service Pack2)

[timfox]

Please post any HornetQ related questions/issues on the HornetQ user forum