HQ122001: Unhandled exception thrown from onMessage: org.infinispan.commons.CacheException: Could not commit implicit transaction
prateekjainaa Dec 11, 2017 5:30 AMWe have 2 jboss instances with replicated/sync infinispan caches. It is intermittently giving OOM error when they are restarted one by one. Here is the exception trace:
2017-12-01 23:37:24,739 WARN [org.infinispan.cache.impl.CacheImpl] (Thread-34 (HornetQ-client-global-threads-1783982683)) ISPN000160: Could not complete injected transaction.: javax.transaction.RollbackException: ARJUNA016053: Could not commit transaction.
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1243)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.infinispan.cache.impl.CacheImpl.tryCommit(CacheImpl.java:1628) [infinispan-core-7.1.1.Final.jar:7.1.1.Final]
at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1585) [infinispan-core-7.1.1.Final.jar:7.1.1.Final]
at org.infinispan.cache.impl.CacheImpl.putInternal(CacheImpl.java:1056) [infinispan-core-7.1.1.Final.jar:7.1.1.Final]
at org.infinispan.cache.impl.CacheImpl.put(CacheImpl.java:1048) [infinispan-core-7.1.1.Final.jar:7.1.1.Final]
at org.infinispan.cache.impl.DecoratedCache.put(DecoratedCache.java:432) [infinispan-core-7.1.1.Final.jar:7.1.1.Final]
at com.blah.cache.infinispan.producer.CacheWrapper.put(CacheWrapper.java:179) [sdk-cache-infinispan-non-cdi-3.48.1.jar:3.48.1]
From heap dump:
Old I/O client worker ([id: 0x1d4e1890, /10.247.246.151:48195 => /10.247.244.26:5445]), runningThread=Thread-48 (HornetQ-client-global-threads-1783982683)
at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)
at org.jboss.netty.buffer.HeapChannelBuffer.<init>(I)V (HeapChannelBuffer.java:42)
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(I)V (BigEndianHeapChannelBuffer.java:34)
at org.jboss.netty.buffer.ChannelBuffers.buffer(Ljava/nio/ByteOrder;I)Lorg/jboss/netty/buffer/ChannelBuffer; (ChannelBuffers.java:134)
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(Ljava/nio/ByteOrder;I)Lorg/jboss/netty/buffer/ChannelBuffer; (HeapChannelBufferFactory.java:68)
at org.jboss.netty.buffer.DynamicChannelBuffer.<init>(Ljava/nio/ByteOrder;ILorg/jboss/netty/buffer/ChannelBufferFactory;)V (DynamicChannelBuffer.java:58)
at org.jboss.netty.buffer.DynamicChannelBuffer.<init>(Ljava/nio/ByteOrder;I)V (DynamicChannelBuffer.java:43)
at org.jboss.netty.buffer.ChannelBuffers.dynamicBuffer(Ljava/nio/ByteOrder;I)Lorg/jboss/netty/buffer/ChannelBuffer; (ChannelBuffers.java:207)
at org.jboss.netty.buffer.ChannelBuffers.dynamicBuffer(I)Lorg/jboss/netty/buffer/ChannelBuffer; (ChannelBuffers.java:197)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.append(Lorg/jboss/netty/buffer/ChannelBuffer;I)V (HornetQFrameDecoder2.java:184)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(Lorg/jboss/netty/channel/ChannelHandlerContext;Lorg/jboss/netty/channel/MessageEvent;)V (HornetQFrameDecoder2.java:94)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(Lorg/jboss/netty/channel/ChannelHandlerContext;Lorg/jboss/netty/channel/ChannelEvent;)V (SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Lorg/jboss/netty/channel/DefaultChannelPipeline$DefaultChannelHandlerContext;Lorg/jboss/netty/channel/ChannelEvent;)V (DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(Lorg/jboss/netty/channel/ChannelEvent;)V (DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Lorg/jboss/netty/channel/Channel;Ljava/lang/Object;Ljava/net/SocketAddress;)V (Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Lorg/jboss/netty/channel/Channel;Ljava/lang/Object;)V (Channels.java:255)
at org.jboss.netty.channel.socket.oio.OioWorker.process()Z (OioWorker.java:71)
at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run()V (AbstractOioWorker.java:73)
at org.jboss.netty.channel.socket.oio.OioWorker.run()V (OioWorker.java:51)
at org.jboss.netty.util.ThreadRenamingRunnable.run()V (ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run()V (DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run()V (VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:615)
at java.lang.Thread.run()V (Thread.java:745)
Any clues what could cause this? We are using Jboss EAP 6.4.0 Feel free to ask in case you need more information.