I was able to get a simple test running with replication of a cache.  One node puts in some data, many other nodes connect, replicate, and can read the data.  When I switch to distributed mode, however, I'm seeing some problems.  I'm sure I'm missing something basic.

My programs are run with -Djgroups.bind_addr=localhost and all nodes are running on the same machine.  I didn't see how I could put that specific piece into the jgroups config file, for me it only seemed to work if I pass it as a JVM argument.  But that's not my real question.

Here's my config:

<infinispan

      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

      xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"

      xmlns="urn:infinispan:config:5.0">

   <global>

      <globalJmxStatistics

            enabled="true"

            jmxDomain="org.infinispan"

            cacheManagerName="SampleCacheManager"/>

      <transport

            clusterName="infinispan-cluster"

            machineId="m1"

            rackId="r1" nodeName="Node-A">

         <properties>

            <property name="configurationFile" value="jgroups.xml" />

         </properties>

      </transport>

   </global>

   <default>

      <locking

         isolationLevel="READ_COMMITTED"

         lockAcquisitionTimeout="20000"

         writeSkewCheck="false"

         concurrencyLevel="5000"

         useLockStriping="false"

      />

      <transaction

            transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"

            syncRollbackPhase="false"

            syncCommitPhase="false"

            useEagerLocking="true"

            eagerLockSingleNode="false"

            cacheStopTimeout="30000" />

      <jmxStatistics enabled="true"/>

      <clustering mode="distribution">

         <sync/>

         <hash

            numOwners="2"

            rehashWait="120000"

            rehashRpcTimeout="600000"

            rehashEnabled="true"

         />

         <l1

            enabled="true"

            lifespan="600000"

         />

      </clustering>

   </default>

</infinispan>

My cache has 10000 elements with 64 char strings as keys and 2K strings as values.

When I'm running my test, I bring up 4 nodes that read a number of random things out of the cache every few seconds and report on successes or failures.  I have just one node that populates the cache, with a 20 ms delay between inserts.  Things seem to start out ok, but after a few minutes I see OutOfMemory errors which I really didn't expect.  Apparently something is going wrong during rebalancing.  This is the message from the node that populated the cache.  After the cache is populated it does nothing at all, just sits there acting as one of the distributed nodes but not modifying or reading the cache directly.

11:27:52,180  INFO JGroupsTransport -- ISPN00094: Received new cluster view: [Node-A-9813|5] [Node-A-9813, Node-A-4857, Node-A-15412, Node-A-19100]

11:28:07,732 ERROR UNICAST2 -- couldn't deliver OOB message [dst: Node-A-19100, src: Node-A-9813 (3 headers), size=60000 bytes, flags=OOB|DONT_BUNDLE|NO_FC]

11:28:13,422 ERROR RebalanceTask -- ISPN00146: Error transferring state to node after rehash

java.util.concurrent.ExecutionException: org.infinispan.CacheException: Problems invoking command.

Caused by: org.infinispan.CacheException: Problems invoking command.

Caused by: java.lang.OutOfMemoryError: Java heap space

On one of the reading nodes I see this set of messages:

11:28:24,803 ERROR Retransmitter -- failed retransmission task

11:28:51,482 ERROR TimeScheduler2 -- failed executing task 148870-148903

11:28:52,471 ERROR TimeScheduler2 -- task execution failed

11:28:51,482  WARN UNICAST2 -- failed sending the message

11:29:04,311 ERROR TCP -- failed sending message to Node-A-15412 (2180 bytes): java.lang.OutOfMemoryError: Java heap space, cause: null

11:29:10,585 ERROR TCP -- failed sending message to Node-A-15412 (2180 bytes): java.lang.OutOfMemoryError: Java heap space, cause: null

11:28:50,588 ERROR UNICAST2 -- couldn't deliver OOB message [dst: Node-A-47, src: Node-A-15412 (3 headers), size=92 bytes, flags=OOB|DONT_BUNDLE|NO_FC]

11:28:46,061 ERROR TCP -- failed sending message to Node-A-15412 (2180 bytes): java.lang.OutOfMemoryError: Java heap space, cause: null

11:28:41,900 ERROR TCP -- failure sending message to 127.0.0.1:7800: java.lang.OutOfMemoryError: Java heap space

11:28:40,467 ERROR jgroups -- uncaught exception in Thread[ConnectionMap.Acceptor,null,null,5,ConnectionMap] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )

11:28:40,467 ERROR TimeScheduler2 -- failed running task org.jgroups.protocols.FD_ALL$TimeoutChecker@c4703a8

11:28:39,880 ERROR UNICAST2 -- couldn't deliver OOB message [dst: Node-A-47, src: Node-A-15412 (3 headers), size=92 bytes, flags=OOB|DONT_BUNDLE|NO_FC]

11:28:37,500 ERROR TCP -- failed handling data from 127.0.0.1:7801

11:29:30,570 ERROR TCP -- failed handling incoming message

11:29:32,370 ERROR jgroups -- uncaught exception in Thread[OOB-63,infinispan-cluster,Node-A-47,5,Thread Pools] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )

11:29:29,713 ERROR TimeScheduler2 -- failed executing task FixedIntervalTask: task=org.jgroups.protocols.FD_ALL$TimeoutChecker@c4703a8, cancelled=false

java.lang.OutOfMemoryError: Java heap space

11:29:24,206 ERROR TCP -- failed sending message to cluster (61 bytes): java.lang.OutOfMemoryError: Java heap space, cause: null

11:29:20,866  WARN UNICAST2 -- failed sending the message

java.lang.OutOfMemoryError: Java heap space

11:29:17,914 ERROR TCP -- failed handling incoming message

11:29:16,993 ERROR Retransmitter -- failed retransmission task

java.lang.OutOfMemoryError: Java heap space

11:29:15,068 ERROR Retransmitter -- failed retransmission task

11:29:46,225 ERROR TimeScheduler2 -- failed executing task 148615-148761

11:29:47,103 ERROR TimeScheduler2 -- task execution failed

11:29:47,847 ERROR jgroups -- uncaught exception in Thread[Timer-6,infinispan-cluster,Node-A-47,5,JGroups] (thread group=org.jgroups.util.Util$1[name=JGroups,maxpri=10] )

11:29:15,068  WARN UNICAST2 -- failed sending the message

11:29:10,992 ERROR RpcManagerImpl -- ISPN00073: Unexpected error while replicating

11:29:10,976  WARN UNICAST2 -- failed sending the message

11:29:53,541 ERROR RequestCorrelator -- failed sending the response

11:29:07,075 ERROR TCP -- failed sending message to Node-A-15412 (2180 bytes): java.lang.OutOfMemoryError: Java heap space, cause: null

11:29:06,027 ERROR TP$TransferQueueBundler -- exception sending bundled msgs: java.lang.OutOfMemoryError: Java heap space:, cause: null

Any ideas what might be causing the OutOfMemory errors? 

Thanks,

Monty