Hello Guys.

tl;dr

     What exactly is causing this or where can I configure the IdleRemover to either not notify the pools which are killing idle connections or do it in a more graceful manner. Flush strategy for all pools is set to FailingConnectionsOnly, so why are inactive connections removed?

long story:

     This is the quick rundown of the problem which I have:

• ActiveMQ brokers are running on few separate machines
• JBoss (running in standalone mode) is hosting the consumers / producers, configured with the failover transport, between the brokers.
• JBoss is using JTA transactions which are binding transactions to PostgreSQL database and ActiveMQ messages.
• ActiveMQ consumers configured as a resource-adapter, with a failover transport between few brokers.
• JCA is binding everything together.

     (parts of the above might not be related to the problem, but just wanted to give you bigger picture)

     The problem was, that in the logs for the JBoss consumers / produces, I could see messages like this:

          13:11:10,469 WARN  [org.apache.activemq.transport.failover.FailoverTransport] (ActiveMQ Transport: tcp://XXXXX@53340) Transport (tcp://YYYYY:31616) failed, reason:  java.io.EOFException, attempting to automatically reconnect

     while the broker was showing (this was displayed as a WARN level message, it's DEBUG in here, because after changing the logging level to see everything, it changed level)

     2014-03-21 12:22:40,486 | DEBUG | Transport Connection to: tcp://XXXXX failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport: tcp:///YYYYY

java.io.EOFException

       at java.io.DataInputStream.readInt(DataInputStream.java:392)

       at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)

       at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)

              at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)

              at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)

     After some digging, I've found out that AMQ was running inactivity checks on it's own - when trying to find root cause of the issue, I've disabled those, so right now the connections should be managed totally by JCA / JBoss. The problem arises because JCA is doing things like this:

      12:39:32,161 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

      12:54:32,191 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

      13:09:32,192 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

      13:24:32,203 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

      13:39:32,255 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

      13:54:32,261 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

      14:09:32,263 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] (IdleRemover) Notifying pools, interval: 900000

     Those are being run every ~15 minutes and inactive AMQ connections are being killed / removed from the pool after 45 minutes.

     My questions are:

• Where / how / what exactly configures that IdleRemover ?
• What are the implications of changing those settings ?
• Shouldn't the connections be removed in a more graceful manner?

As written in the beginning - any help in here is appreciated.

Thanks you kindly,

-- michal.