5 Replies Latest reply on May 20, 2015 3:11 AM by Horia Chiorean

    Infinispan OutOfMemoryError when transferring state in a WildFly cluster

    Ryan Weber Newbie

      Running ModeShape 4.1 as subsystem on WildFly 8.2.0.Final, cluster of two nodes (A and B). On each node, ModeShape is configured with two replicated caches for a binary store (one for meta data), each backed by a file store -- see below.

       

              <subsystem xmlns="urn:jboss:domain:infinispan:2.0">
                  ...
                  <cache-container name="modeshape" module="org.modeshape">
                      <transport lock-timeout="60000"/>
                      <replicated-cache name="myrepository" batching="false" mode="SYNC">
                          <transaction mode="NON_XA"/>
                          <file-store passivation="false" purge="false" relative-to="jboss.server.data.dir" path="modeshape/store/myrepository"/>
                      </replicated-cache>
                  </cache-container>
                  <cache-container name="modeshape-binary-store" module="org.modeshape">
                      <transport lock-timeout="60000"/>
                      <replicated-cache name="myrepository-binary-data" batching="false" mode="SYNC">
                          <transaction mode="NON_XA"/>
                          <file-store passivation="false" purge="false" relative-to="jboss.server.data.dir" path="modeshape/binary-store/myrepository"/>
                      </replicated-cache>
                      <replicated-cache name="myrepository-binary-metadata" batching="false" mode="SYNC">
                          <transaction mode="NON_XA"/>
                          <file-store passivation="false" purge="false" relative-to="jboss.server.data.dir" path="modeshape/binary-store/myrepository-metadata"/>
                      </replicated-cache>
                  </cache-container>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:modeshape:2.0">
                  <repository name="myrepository" cache-name="myrepository" cache-container="modeshape">
                      <cache-binary-storage data-cache-name="myrepository-binary-data" metadata-cache-name="myrepository-binary-metadata" cache-container="modeshape-binary-store"/>
                      <workspaces allow-workspace-creation="false">
                          <workspace name="default"/>
                      </workspaces>
                  </repository>
              </subsystem>

       

      First, the cache container on Node A starts when it is accessed. Then, the following exception occurs on Node A when the Node B cache container is accessed and starts:

       

      2015-04-13 23:05:01,460 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-11) Failed to send entries to node A/modeshape-binary-store : java.lang.OutOfMemoryError: Java heap space: org.infinispan.commons.CacheException: java.lang.OutOfMemoryE
      rror: Java heap space
              at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:294) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:233) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:174) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_75]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_75]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_75]
              at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75]
              at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
      Caused by: java.lang.OutOfMemoryError: Java heap space
              at org.infinispan.commons.io.ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:71)
              at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:108)
              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:300)
              at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
              at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
              at org.infinispan.container.entries.metadata.MetadataImmortalCacheEntry$Externalizer.writeObject(MetadataImmortalCacheEntry.java:58) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.container.entries.metadata.MetadataImmortalCacheEntry$Externalizer.writeObject(MetadataImmortalCacheEntry.java:54) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
              at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
              at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
              at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:25)
              at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:44) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:26) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
              at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
              at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
              at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:80) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:65) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
              at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
              at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
              at org.infinispan.commons.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:25)
              at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:44) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.marshall.exts.ListExternalizer.writeObject(ListExternalizer.java:26) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:395) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
              at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:138)
              at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:58)
              at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:111)
              at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:57) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]

       

      This configuration was working last week, but more files were added to the repository. However, at present, the repository contains no more than 50 files total, each less than 3 MB.

       

      Increasing the JVM heap size to 3 GB (from the defaults) on each node resolves the issue for now. I am concerned that this exception will occur again when the repository has grown to a certain size.