1 Reply Latest reply on Dec 31, 2014 4:16 AM by Horia Chiorean

    Trouble making replication work

    Ross Woolf Newbie

      I am unable to get replication with Modeshape working.  I would appreciate it if someone could point me in the right direction as to how to get things working.

      I have created a web application running in Tomcat. I am using ModeShape 3.7.1.Final.

      Using the jgroups configuration provided in the documentation at ModeShape in Java applications - ModeShape 3 - Project Documentation Editor I get the following errors (I also used the information from this page to configure replication).

      In the tomcat stderr log it states:

      org.modeshape.common.SystemFailureException: Error deserializing changes obtained from channel 'my-repo-cluster'

        at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus$Receiver.receive(ClusteredRepositoryChangeBus.java:316)

        at org.jgroups.JChannel.invokeCallback(JChannel.java:749)

        at org.jgroups.JChannel.up(JChannel.java:710)

        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)

        at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:178)

        at org.jgroups.protocols.RSVP.up(RSVP.java:188)

        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)

        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)

        at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:599)

        at org.jgroups.protocols.UNICAST.up(UNICAST.java:399)

        at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606)

        at org.jgroups.protocols.BARRIER.up(BARRIER.java:102)

        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)

        at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187)

        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)

        at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

        at org.jgroups.protocols.Discovery.up(Discovery.java:359)

        at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)

        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)

        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

      Caused by: java.io.StreamCorruptedException: invalid stream header: 0301FE03

        at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)

        at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)

        at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus$ObjectInputStreamWithClassLoader.<init>(ClusteredRepositoryChangeBus.java:367)

        at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus.deserialize(ClusteredRepositoryChangeBus.java:287)

        at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus$Receiver.receive(ClusteredRepositoryChangeBus.java:309)

        ... 25 more

       

      In the stdout log it states:

      12:50:19.813 [localhost-startStop-1] ERROR o.i.i.InvocationContextInterceptor - ISPN000136: Execution error

      org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to SVP321APP-30085

        at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.CacheStoreInterceptor.visitPrepareCommand(CacheStoreInterceptor.java:208) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) [jbossjta-4.16.6.Final.jar:na]

        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) [jbossjta-4.16.6.Final.jar:na]

        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) [jbossjta-4.16.6.Final.jar:na]

        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164) [jbossjta-4.16.6.Final.jar:na]

        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) [jbossjta-4.16.6.Final.jar:na]

        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) [jbossjta-4.16.6.Final.jar:na]

        at org.modeshape.jcr.txn.Transactions$SimpleTransaction.commit(Transactions.java:296) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:472) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:420) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.SystemContent.save(SystemContent.java:111) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.SystemNamespaceRegistry.register(SystemNamespaceRegistry.java:179) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1137) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:971) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:404) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:648) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:619) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:155) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]

        at com.tybera.dms.modeShape.DmsModeShapeEngine.getNewSession(DmsModeShapeEngine.java:211) [DmsModeShapeEngine.class:na]

        at com.tybera.dms.modeShape.DmsModeShapeEngine.setFileFields(DmsModeShapeEngine.java:184) [DmsModeShapeEngine.class:na]

        at com.tybera.dms.modeShape.DmsModeShapeEngine.startEngine(DmsModeShapeEngine.java:151) [DmsModeShapeEngine.class:na]

        at com.tybera.dms.modeShape.DmsModeShapeEngine.init(DmsModeShapeEngine.java:79) [DmsModeShapeEngine.class:na]

        at com.tybera.dms.servlets.AppContextListener.contextInitialized(AppContextListener.java:22) [AppContextListener.class:na]

        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939) [catalina.jar:7.0.47]

        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434) [catalina.jar:7.0.47]

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:7.0.47]

        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) [catalina.jar:7.0.47]

        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) [catalina.jar:7.0.47]

        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633) [catalina.jar:7.0.47]

        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:983) [catalina.jar:7.0.47]

        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1660) [catalina.jar:7.0.47]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_60]

        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]

        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]

      Caused by: org.jgroups.TimeoutException: timeout sending message to SVP321APP-30085

        at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392) ~[jgroups-3.2.10.Final.jar:3.2.10.Final]

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]

        ... 84 common frames omitted

       

      My configuration files are as follows:

      Infinispan configuration:

      <infinispan xmlns="urn:infinispan:config:5.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd">

          <global>

          <globalJmxStatistics allowDuplicateDomains="true"/>

        <!--<transport machineId="192.168.128.50" clusterName="my-repo-cluster" nodeName="linux">-->

        <transport clusterName="my-repo-cluster">

         <properties>

           <property name="configurationFile" value="jgroups-config.xml"/>

         </properties>

        </transport>

          </global>

          <default>

            <clustering mode="replication"> <sync/> </clustering>

      </default>

      <namedCache name="persisted-repository">

          <eviction strategy="LIRS" maxEntries="600"/>

          <!-- Configure a synchronous replication cache -->

          <clustering mode="replication">

            <sync/>

          </clustering>

          <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" transactionMode="TRANSACTIONAL" lockingMode="PESSIMISTIC"/>

          <loaders passivation="false" shared="false" preload="false">

            <loader class="org.infinispan.loaders.jdbm.JdbmCacheStore" fetchPersistentState="true" purgeOnStartup="false">

              <properties>

                <property name="location" value="D:\test\repository\longterm\content"/>

              </properties>

            </loader>

          </loaders>

        </namedCache>

      </infinispan>

       

      ModeShape configuration:

      {

        "name" : "longterm",

        "monitoring" : {

          "enabled" : false

        },

        "workspaces" : {

          "predefined" : [ "otherWorkspace" ],

          "default" : "default",

          "allowCreation" : true

        },

        "security" : {

          "anonymous" : {

            "roles" : [ "readonly", "readwrite", "admin" ],

            "useOnFailedLogin" : false

          }

        },

        "storage" : {

          "cacheConfiguration" : "D:\\test\\conf\\longterm-infinispan-configuration.xml",

          "cacheName" : "persisted-repository",

          "binaryStorage" : {

            "type" : "file",

            "directory" : "D:\\test\\repository\\longterm\\binaries",

            "minimumBinarySizeInBytes" : 999

          }

        },

        "query" : {

          "enabled" : true,

          "indexStorage" : {

            "type" : "filesystem",

            "location" : "D:\\test\\repository\\longterm\\indexes",

            "lockingStrategy" : "simple",

            "fileSystemAccessType" : "auto"

          },

          "textExtracting" : {

            "extractors" : {

              "tikaExtractor" : {

                "name" : "General content-based extractor",

                "classname" : "tika"

              }

            }

          },

          "indexing" : {

            "rebuildOnStartup" : {

              "when" : "if_missing",

              "mode" : "async"

            }

          }

        },

        "clustering" : {

          "clusterName" : "my-repo-cluster",

          "channelConfiguration" : "jgroups-config.xml"

        }

      }

       

      If I remove the configuration to use the "jgroups-config.xml" from both of these configuration files so that it will use the jgroups default configuration then I no longer get the errors logged in the log files, but I now see the following warning in the stderr log which is constantly repeated.

      Dec 30, 2014 12:11:14 PM org.jgroups.logging.JDKLogImpl warn

      WARNING: SVP321APP-53335: no physical address for 867214b2-c0af-a43e-4792-9e748f12801f, dropping message

       

      Running in the default jgroups configuraiton I can use wireshark and see that messages are sent from each instance to the other instance so it appears there is some communication between the two instances.  But when I add something to one instance of modeshape it never gets replicated. 

       

      As an added test I simply ran the jgroups draw demo on each of the servers as described here: http://www.jgroups.org/tutorial/index.html.  Rather than running two instances on the same server, I ran one instance on each server and as I draw on one I can see the results on the other server.  So I know that jgroups in its default configuration is working as far as this demo app is concerned.

       

      The frustrating thing is that whatever I have tried has not worked to cause any replication to occur.  It is almost as if ModeShape is not even attempting to do any replication.  I see nothing in the logs that it is even attempting to do anything.  All I see is the wireshark communication between the instances of JGroups, but in none of that communication do I see anything regarding replicating data.

       

      At this point I don't have any idea on what to try or how to debug the problem of no replication.  I would appreciate if anyone can help me understand what I am missing, or where I can get further info as to what may be going wrong, or how to debug this problem further.

       

      Thanks.

        • 1. Re: Trouble making replication work
          Horia Chiorean Master

          Before going into details, if possible you should always try using the latest community versions (which is 4.1 for 4.x and 3.8.1 for 3.x) since we're always fixing different bugs. Also, 3.x is not supported in the community anymore since we've moved to 4. Clustering configuration has greatly changed (and simplified) in 4 compared to 3.

           

          In 3.x, clustering happens on 2 different levels:

          1. infinispan cache level (sending the actual nodes & data across the wire) - controlled entirely via the Infinispan configuration

          2. repository level (sending repository events across the wire - essentially notifying other nodes that data has changed) - controlled via the clustering section in the JSON configuration

           

          From the stack traces you posted above, it seems neither of those levels are working correctly (ISPN replication is timing out while ModeShape event replication is failing with StreamCorruptedException). So I think you're right that this might be some sort of JGroups configuration issue for your particular case. I'm no JGroups expert, so unfortunately there's not much more I can tell you. For what it's worth, in 3.x we're using this configuration modeshape/jgroups-test-config.xml at 3.x · ModeShape/modeshape · GitHub for our unit tests which run locally (binding to 127.0.0.1 or ::1 and using UDP) and which works fine in that case.

           

          You should try setting the log level of org.infinispan TRACE and see if anything more meaningful comes up. You can also use the Infinispan forms for questions regarding JGroups configuration/ISPN replication.