We're using Jboss EAP 6.4, with a chain to handle jms messages scenario of:
- Jboss EAP 6.4.U12
- jbossjts 4.17.38 as transaction manager
- ironjacamar 1.0.37 as java connector
- com.stc.jmsjca.core-6.2.0 as resource adapter
- and wlthint3client.jar as weblogic drivers.
- Weblogic 12c , as JMS external provider.
The jms flow is quite fine , despite few transactions that are timed out, on which we're investigating.
We're experiencing a problem with Periodic Recovery.
Just to summarize, seems that a bunch of JMSXAConnectionFactory, JMSXAConnection and JMSSession objects are opened by recovery manager, during PeriodicRecovery.run method, are not getting closed, even at the end of the Periodic Recovery work, eventually causing the filling of the heap of the JVM after a while.
We're crossing Periodic Recovery log files ( com.arjuna is logging to TRACE on an APPENDER ) , sampling objects with jmap -histo:live,
noticing that at every run of the Periodic Recovery, those JMSXAConnectionFactory, the JMSXAConnection and JMSSessions are being allocated, and never released.
Studying the doc http://narayana.io/docs/product/#d0e826 -> 2.1.4. A Recovery Module for XA Resources, and trying to follow also in the code, so far we noticed that the connections are getting allocated when XARecoveryModule class launch its private method "resourceInitiatedRecoveryForRecoveryHelpers".
There with the help of XAResourceRecoveryHelperWrapper the method "getXAResources ()" declared by the interface org.jboss.tm.XAResourceRecovery is being called . This get us to ironjacamar , in the interface implementation XAResourceRecoveryInflowImpl . That classes finally has got the logic to deal with the resource adapter.
In our case the method getXAResources of the class com.stc.jmsjca.core.RAJMSResourceAdapter is being called.
There, on a RAJMSObjectFactory object , the resource adapter creates the JMSXAConnectionFactory, the JMSXAConnection and JMSSession, in order to return the caller the proper XAresources, on which Xid will be recoverd.. At the end of the getXAResources(), a list of "recovery connections" is populated,
but we're able to see only a private method ( checkRecoveryConnections() ), that can deal with that object. This will handle the explicit closure of the connection resources.
Then we're able to view Periodic Recovery and XARecoveryModule that work with those XAresources returned,
completing xaRecoveryFirstPass and the periodicWorkSecondPass.
But at the end of the periodic recovery, we were not able to catch an explicit closure of those connection objects (JMSXAConnectionFactory, JMSXAConnection , JMSSession), spawned by ResourceAdapter. As a result, the connection nor getting reused neither closed, but at each run of Periodic Recovery, a bunch of brand new connections are being opened, again and again.
My questions are:
- Who is supposed to be in charge to close those objects ? (The Container ? The Periodic Recovery ? The resource adapter ?
- Could be an eventual configuration fault (ra.xml or ironjacamar.xml or again jboss standalone.xml)
As a counter proof, if the PeriodicRecovery is disable by jvm system option, the leaked connections objects are not being created.
The normal connection objects spawned by producers and consumers remain stable.
Please let me if I could help with any config details.
Thanks in advance ,