It doesn't seem right that I need to use XA transactions. A commit is a commit and should always work fully or not work. XA transactions are for multiple systems which this is not. I just do a single commit and the server commits while the client fails. This is a bug for sure.
I forgot to mention. I'm retrieving 500 messages per transaction commit.
It seems that retrieving less messages and increasing timeout does go some way to alleviating the problem.
By the way, the server keeps logging the messages like this:
[hornetq-failure-check-thread] 13:09:11,189 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49686. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 13:09:11,698 WARNING [org.hornetq.core.server.impl.ServerSessionImpl] Client connection failed, clearing up resources for session f0885f06-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:11,773 WARNING [org.hornetq.core.server.impl.ServerSessionImpl] Cleared up resources for session f0885f06-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:11,774 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Client connection failed, clearing up resources for session f0885f06-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:11,847 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Cleared up resources for session f0885f06-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,461 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49687. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 13:09:12,462 WARNING [org.hornetq.core.server.impl.ServerSessionImpl] Client connection failed, clearing up resources for session f2b06f29-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,467 WARNING [org.hornetq.core.server.impl.ServerSessionImpl] Cleared up resources for session f2b06f29-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,468 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Client connection failed, clearing up resources for session f2b06f29-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,715 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Cleared up resources for session f2b06f29-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,787 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49857. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 13:09:12,788 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Client connection failed, clearing up resources for session a6d86df0-79e8-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,834 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Cleared up resources for session a6d86df0-79e8-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,835 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Client connection failed, clearing up resources for session a6e0ab51-79e8-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,837 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Cleared up resources for session a6e0ab51-79e8-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,844 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49683. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 13:09:12,845 WARNING [org.hornetq.core.server.impl.ServerSessionImpl] Client connection failed, clearing up resources for session eeede8e3-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,846 WARNING [org.hornetq.core.server.impl.ServerSessionImpl] Cleared up resources for session eeede8e3-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,846 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Client connection failed, clearing up resources for session eeede8e3-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:09:12,849 WARNING [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Cleared up resources for session eeede8e3-79e7-11e0-aa1c-001fc6ffeb61
[hornetq-failure-check-thread] 13:10:12,900 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49907. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 13:10:14,903 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49909. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
[hornetq-failure-check-thread] 13:10:14,937 WARNING [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /192.168.196.121:49908. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3]
So the server is obviously very confused even though it is causing the problem and the client does a full disconnect/reconnect.