Xa - Memory Leak ?
We are using 1.0.1SP4 on JBoss405.
The queues are persisted on oracle 10g.
JBM has its own non-xa oracle datasource
The multithreaded consumers rely on the XA Connection Factory and are wrapped in XA transactions.
One set of connection/session/consumer is created for each sequence of polling, no caching whatsoever, and we proceed with the usual pattern :
xaTx.begin
con = jmxXACf.createCon
ses = con.createSession
con.start
consumer = ses.createConsumer
consumer.receive()
consumer.close
session.close
con.close
xaTx.commit
they're all cleaned up after the receive call, and the XA transaction is always commited/rollbacked.
The jboss instance hosting both JBM and the consumers (bootstraped from a War using JBM loader) runs slowly but steadily to memory and cpu exhaustion, even when it's activity is limited to the constant jms 'polling' (8 consumer threads).
As the heap graphs were showing textbook linear memory consumption I ran jmap/jhat after a 2 hours run (with only the 30th first minutes of dense activity). It showed that the number of ServerConsumerEndpoint instances had made the top 5 with approx 200000 :
num #instances #bytes class name
--------------------------------------
1: 217640 42500368 [Ljava.util.HashMap$Entry;
2: 223956 25354776 [Ljava.lang.Object;
3: 306257 23970568 [C
4: 468932 22508736 java.util.HashMap$Entry
5: 199658 20764432 org.jboss.jms.server.endpoint.ServerConsumerEndpoint
Then I get all kind of stacks bound to the memory shortage like :
java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:50:39,420 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] - could not create session org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection@3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri@2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException) at org.jboss.resource.JBossResourceException.rethrowAsResourceException(JBossResourceException.java:61) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:410) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:389) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369) at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.reflect.UndeclaredThrowableException at org.jboss.resource.JBossResourceException.process(JBossResourceException.java:204) at org.jboss.resource.JBossResourceException.<init>(JBossResourceException.java:111) ... 12 more Caused by: java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "Timer-1" 2007-02-09 10:51:13,748 ERROR [STDERR] - Exception in thread "TimeoutFactory-0" 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:13,748 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:24,326 ERROR [org.jboss.jms.util.ExceptionUtil] - ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s] java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:38,514 ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483608] close [a1885-6t7h9a-exxk6yht-1-exyfx05u-3t] java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:41,951 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:55,701 ERROR [org.jboss.jms.util.ExceptionUtil] - SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u] java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:55,717 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:59,170 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:51:59,170 ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:52:02,623 INFO [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection 2007-02-09 10:52:06,061 ERROR [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Setup of JMS message listener invoker failed - trying to recover java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:52:19,858 INFO [com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer] - Successfully refreshed JMS Connection 2007-02-09 10:52:23,326 ERROR [cms.transcoding.technical] - Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection@3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri@2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException) javax.jms.JMSException: Could not create a session: org.jboss.resource.JBossResourceException: Unchecked throwable in ManagedConnection.getConnection() cl=org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1a39aa42[state=NORMAL mc=org.jboss.resource.adapter.jms.JmsManagedConnection@3f9c2783 handles=0 lastUse=1171014618826 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$PoolBySubjectAndCri@2e2722df context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@b627625 xaResource=MessagingXAResource[-2147483607] txSync=null]; - nested throwable: (java.lang.reflect.UndeclaredThrowableException) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:403) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createSession(JmsSessionFactoryImpl.java:369) at org.springframework.jms.listener.AbstractMessageListenerContainer.createSession(AbstractMessageListenerContainer.java:1002) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:360) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) at java.lang.Thread.run(Thread.java:595) 2007-02-09 10:52:33,920 ERROR [org.jboss.jms.client.container.ExceptionInterceptor] - Caught Error: java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:52:48,030 ERROR [STDERR] - Exception in thread "JCA PoolFiller" 2007-02-09 10:52:48,030 ERROR [STDERR] - java.lang.OutOfMemoryError: Java heap space 2007-02-09 10:53:06,014 ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s] org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. ConnectionEndpoint[-2147483643] start [a1885-6t7h9a-exxk6yht-1-exyfwp7q-3s] at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72) at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.start(ServerConnectionEndpoint.java:254) at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$start$aop(ConnectionAdvised.java:94) at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723) at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014) at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98) at org.jboss.remoting.Client.invoke(Client.java:607) at org.jboss.remoting.Client.invoke(Client.java:599) at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeNext(ClientConnectionDelegate$start_N8025343665958530775.java) at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java) at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121) at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:599) at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:530) at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:239) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:361) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) at java.lang.Thread.run(Thread.java:595) 2007-02-09 10:54:39,248 ERROR [cms.transcoding.technical] - A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u] org.jboss.jms.util.MessagingJMSException: A failure has occurred during processing of the request. Please consult the server logs for more details. SessionEndpoint[-2147483602] createConsumerDelegate [a1885-6t7h9a-exxk6yht-1-exyfxarc-3u] at org.jboss.jms.util.ExceptionUtil.handleJMSInvocation(ExceptionUtil.java:72) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:287) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:90) at sun.reflect.GeneratedMethodAccessor137.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.jms.server.container.SecurityAspect.handleCreateConsumerDelegate(SecurityAspect.java:116) at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:130) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aop.Advisor.dynamicInvoke(Advisor.java:723) at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:101) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1014) at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98) at org.jboss.remoting.Client.invoke(Client.java:607) at org.jboss.remoting.Client.invoke(Client.java:599) at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:122) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect20.invoke(StateCreationAspect20.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:69) at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect19.invoke(ConsumerAspect19.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_6311124154581125663.invokeNext(ClientSessionDelegate$createConsumerDelegate_6311124154581125663.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:254) at org.jboss.jms.client.JBossSession.createConsumer(JBossSession.java:231) at org.jboss.resource.adapter.jms.JmsSession.createConsumer(JmsSession.java:473) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createConsumer(DestinationBalancingMessageListenerContainer.java:648) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.createListenerConsumer(DestinationBalancingMessageListenerContainer.java:302) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.doReceiveAndExecute(DestinationBalancingMessageListenerContainer.java:363) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer.receiveAndExecute(DestinationBalancingMessageListenerContainer.java:326) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DestinationBalancingMessageListenerContainer.java:716) at com.digiplug.repository.transcoding.util.spring.DestinationBalancingMessageListenerContainer$AsyncMessageListenerInvoker.run(DestinationBalancingMessageListenerContainer.java:680) at org.springframework.core.task.SimpleAsyncTaskExecutor$ConcurrencyThrottlingRunnable.run(SimpleAsyncTaskExecutor.java:203) at java.lang.Thread.run(Thread.java:595)
I saw that some XA related memory leaks issues had been fixed into SP4, so I wondered if these elements might ring a bell before I dig further down...
This scenario is very easy to reproduce as-is on production,preprod or dev environments, I didnt have the time yet to create a barebone test, Id rather not write several and Im not sure how much to cut (xa, oracle, spring ?)
This not a critical issue as the server handles perfectly the load peaks during the first hour(s) of its (short) life, and we can restart it easily
but it's obviously a far-from-ideal situation.
Thanks for your help,
Best Regards
Olivier Cuzacq
Digiplug.com
