Infinispan OutOfMemoryError when transferring state in a WildFly cluster
ryanweber Apr 14, 2015 1:36 PMRunning ModeShape 4.1 as subsystem on WildFly 8.2.0.Final, cluster of two nodes (A and B). On each node, ModeShape is configured with two replicated caches for a binary store (one for meta data), each backed by a file store -- see below.
<subsystem xmlns="urn:jboss:domain:infinispan:2.0"> ... <cache-container name="modeshape" module="org.modeshape"> <transport lock-timeout="60000"/> <replicated-cache name="myrepository" batching="false" mode="SYNC"> <transaction mode="NON_XA"/> <file-store passivation="false" purge="false" relative-to="jboss.server.data.dir" path="modeshape/store/myrepository"/> </replicated-cache> </cache-container> <cache-container name="modeshape-binary-store" module="org.modeshape"> <transport lock-timeout="60000"/> <replicated-cache name="myrepository-binary-data" batching="false" mode="SYNC"> <transaction mode="NON_XA"/> <file-store passivation="false" purge="false" relative-to="jboss.server.data.dir" path="modeshape/binary-store/myrepository"/> </replicated-cache> <replicated-cache name="myrepository-binary-metadata" batching="false" mode="SYNC"> <transaction mode="NON_XA"/> <file-store passivation="false" purge="false" relative-to="jboss.server.data.dir" path="modeshape/binary-store/myrepository-metadata"/> </replicated-cache> </cache-container> </subsystem> <subsystem xmlns="urn:jboss:domain:modeshape:2.0"> <repository name="myrepository" cache-name="myrepository" cache-container="modeshape"> <cache-binary-storage data-cache-name="myrepository-binary-data" metadata-cache-name="myrepository-binary-metadata" cache-container="modeshape-binary-store"/> <workspaces allow-workspace-creation="false"> <workspace name="default"/> </workspaces> </repository> </subsystem>
First, the cache container on Node A starts when it is accessed. Then, the following exception occurs on Node A when the Node B cache container is accessed and starts:
2015-04-13 23:05:01,460 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-11) Failed to send entries to node A/modeshape-binary-store : java.lang.OutOfMemoryError: Java heap space: org.infinispan.commons.CacheException: java.lang.OutOfMemoryE rror: Java heap space at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:294) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:233) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:174) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_75] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_75] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_75] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75] Caused by: java.lang.OutOfMemoryError: Java heap space at org.infinispan.commons.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:71) at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:108) at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:300) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.container.entries.metadata.MetadataImmortalCacheEntry$Externalizer.writeObject(MetadataImmortalCacheEntry.java:58) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.container.entries.metadata.MetadataImmortalCacheEntry$Externalizer.writeObject(MetadataImmortalCacheEntry.java:54) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:25) at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:44) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:26) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:80) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:65) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:25) at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:44) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:26) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final] at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138) at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58) at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111) at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
This configuration was working last week, but more files were added to the repository. However, at present, the repository contains no more than 50 files total, each less than 3 MB.
Increasing the JVM heap size to 3 GB (from the defaults) on each node resolves the issue for now. I am concerned that this exception will occur again when the repository has grown to a certain size.