memory leak in arjuna TransactionImple?
Hello Community,
acutally i'm facing a huge problem: the memory used by my jboss increases permanently during heavy usage.
The JBoss EAP 6.4.1 has an Heap of 19G and a PermGenSpace of 512M running on JDK 1.7_45.
G1 is enabled as GarbageCollector, Eden Space is cofigured to min. 45%. So we have an OldGen of about 10.700M.
Starting the server, and working for several ours uses about 4G of space in OldGen.
During further usage, the amount used by the OldGen grows by about 1G till 2G per workday (heavy usage).
After at least 5 workdays the OldGen uses about 9G. The the GC got allocation errors during peaks and do a Full GC. This repeats for about 10 times but the free space in OldGen after a Full GC melts till zero and the distance between Full GC gets smaller. Working with the System isn't possible at this time, so the only thing that helps is to do a restart.
After intensive checking for bad configuration in case of hibernate 2nd level cache or own application code i excluded these as possible issues.
So we took a HeapDump during the phase of full OldGen and the result ist, that about 70% of OldGen space is blocked by Objects referenced by the static HashMap "_transactions" from TransactionImple (com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple).
(see attached screenshot)
During a code review of TransactionImple it is suspect to me, that the only way to remove transactions is by calling commit() or rollback() having the transaction in one of the states "RUNNING" or "ABORT_ONLY".
Otherwise it will throw an exception before removing the transaction from the map.
Having our Heapdump and looking into the transactions hold by this map, they seem all to have status 4 (== ABORTED).
We also sometime faces transaction timeouts, so we first see logging of TransactionReaper Thread and then "The transaction is not active! Uid is ..."
Another way transactions are crashing with oracle sql exceptions during commit, so the transaction is in state 6 (==COMMITTING) during making an rollback.
Does anyone else think this could be a possible memory leak? I would like to ask the community befor raising an unnessesary issue.
Some infos to our application:
The jboss just provides backround functionalities using enterprise beans and does not host any frontends like jsf. There are also beans that are triggered from interal, for exaplme by jms messages or timers.
All real clients acessing the beans via ejb remoting.
We are using Jboss EAP 6.4.1 so we have jbossjts-jacorb-4.17.29.Final-redhat-1.jar
At the moment we have following bean amounts:
1.130 stateless EJB
0 stateful EJB
61 singleton EJB
38 MessageDriven EJB
Our slsb-strict-max-pool has max size 40.
Our mdb-strict-max-pool has max size 15.
24 of our beans has bean managed transaction, all others are container managed.
