We are running HornetQ in standalone (no jboss) HA mode. We see an issue when clients are forcibly closed (kill -9). The HornetQ appears to detect the connection failure and clean up the connections, however the sessions do not always appear to be removed. These phantom sessions within hornetq server continue to receive messages resulting in topics eventually filling causing producers to block.
Any ideas on what this could be? Are there any known issues with connection failure detection?
From some custom management beans we can see that the connection is removed from the serverManager.getHornetQServer().getRemotingService().getConnections() but the corresponding session remains in serverManager.getHornetQServer().getSessions().
The server and (clients jars) are built from Tag HornetQ_2_2_24_EAP_GA. (We saw the same issue with 2.2.14.Final)
[hornetq-failure-check-thread] 19-Feb 18:10:59,738 WARNING [RemotingConnectionImpl] Connection failure has been detected: Did not receive data from /10.32.50.151:56483. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 19-Feb 18:10:59,738 WARNING [ServerSessionImpl] Client connection failed, clearing up resources for session 26b058e7-7a5b-11e2-b783-7399b1160711
[hornetq-failure-check-thread] 19-Feb 18:10:59,739 WARNING [ServerSessionImpl] Cleared up resources for session 26b058e7-7a5b-11e2-b783-7399b1160711
[hornetq-failure-check-thread] 19-Feb 18:10:59,740 WARNING [ServerSessionImpl] Client connection failed, clearing up resources for session 26af6e84-7a5b-11e2-b783-7399b1160711
[hornetq-failure-check-thread] 19-Feb 18:10:59,743 WARNING [ServerSessionImpl] Cleared up resources for session 26af6e84-7a5b-11e2-b783-7399b1160711
Th is appears to have been caused by a client application running in Jboss 6.1.0 which was using older version of the hornetq client libraries. (jboss 6.1.0 is packaged with HornetQ 2.2.5). Once the hornetq client libraries in jboss were updated to 2.2.24, the issue went away.