Trouble making replication work
I am unable to get replication with Modeshape working. I would appreciate it if someone could point me in the right direction as to how to get things working.
I have created a web application running in Tomcat. I am using ModeShape 3.7.1.Final.
Using the jgroups configuration provided in the documentation at ModeShape in Java applications - ModeShape 3 - Project Documentation Editor I get the following errors (I also used the information from this page to configure replication).
In the tomcat stderr log it states:
org.modeshape.common.SystemFailureException: Error deserializing changes obtained from channel 'my-repo-cluster'
at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus$Receiver.receive(ClusteredRepositoryChangeBus.java:316)
at org.jgroups.JChannel.invokeCallback(JChannel.java:749)
at org.jgroups.JChannel.up(JChannel.java:710)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1025)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:178)
at org.jgroups.protocols.RSVP.up(RSVP.java:188)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:896)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:245)
at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:599)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:399)
at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:606)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:102)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:187)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)
at org.jgroups.protocols.Discovery.up(Discovery.java:359)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1263)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1825)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1798)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.StreamCorruptedException: invalid stream header: 0301FE03
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus$ObjectInputStreamWithClassLoader.<init>(ClusteredRepositoryChangeBus.java:367)
at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus.deserialize(ClusteredRepositoryChangeBus.java:287)
at org.modeshape.jcr.bus.ClusteredRepositoryChangeBus$Receiver.receive(ClusteredRepositoryChangeBus.java:309)
... 25 more
In the stdout log it states:
12:50:19.813 [localhost-startStop-1] ERROR o.i.i.InvocationContextInterceptor - ISPN000136: Execution error
org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to SVP321APP-30085
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:247) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:234) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:208) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:109) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:101) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.CacheStoreInterceptor.visitPrepareCommand(CacheStoreInterceptor.java:208) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:106) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:108) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:103) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:128) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:115) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.TransactionSynchronizerInterceptor.visitPrepareCommand(TransactionSynchronizerInterceptor.java:61) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:189) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:93) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:124) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:175) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96) [jbossjta-4.16.6.Final.jar:na]
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402) [jbossjta-4.16.6.Final.jar:na]
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103) [jbossjta-4.16.6.Final.jar:na]
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164) [jbossjta-4.16.6.Final.jar:na]
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165) [jbossjta-4.16.6.Final.jar:na]
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117) [jbossjta-4.16.6.Final.jar:na]
at org.modeshape.jcr.txn.Transactions$SimpleTransaction.commit(Transactions.java:296) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:472) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:420) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.SystemContent.save(SystemContent.java:111) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.SystemNamespaceRegistry.register(SystemNamespaceRegistry.java:179) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1137) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:971) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:404) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:648) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:619) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:155) [modeshape-jcr-3.7.1.Final.jar:3.7.1.Final]
at com.tybera.dms.modeShape.DmsModeShapeEngine.getNewSession(DmsModeShapeEngine.java:211) [DmsModeShapeEngine.class:na]
at com.tybera.dms.modeShape.DmsModeShapeEngine.setFileFields(DmsModeShapeEngine.java:184) [DmsModeShapeEngine.class:na]
at com.tybera.dms.modeShape.DmsModeShapeEngine.startEngine(DmsModeShapeEngine.java:151) [DmsModeShapeEngine.class:na]
at com.tybera.dms.modeShape.DmsModeShapeEngine.init(DmsModeShapeEngine.java:79) [DmsModeShapeEngine.class:na]
at com.tybera.dms.servlets.AppContextListener.contextInitialized(AppContextListener.java:22) [AppContextListener.class:na]
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4939) [catalina.jar:7.0.47]
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5434) [catalina.jar:7.0.47]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) [catalina.jar:7.0.47]
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) [catalina.jar:7.0.47]
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) [catalina.jar:7.0.47]
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633) [catalina.jar:7.0.47]
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:983) [catalina.jar:7.0.47]
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1660) [catalina.jar:7.0.47]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_60]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: org.jgroups.TimeoutException: timeout sending message to SVP321APP-30085
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:392) ~[jgroups-3.2.10.Final.jar:3.2.10.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179) ~[infinispan-core-5.2.7.Final.jar:5.2.7.Final]
... 84 common frames omitted
My configuration files are as follows:
Infinispan configuration:
<infinispan xmlns="urn:infinispan:config:5.1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd">
<global>
<globalJmxStatistics allowDuplicateDomains="true"/>
<!--<transport machineId="192.168.128.50" clusterName="my-repo-cluster" nodeName="linux">-->
<transport clusterName="my-repo-cluster">
<properties>
<property name="configurationFile" value="jgroups-config.xml"/>
</properties>
</transport>
</global>
<default>
<clustering mode="replication"> <sync/> </clustering>
</default>
<namedCache name="persisted-repository">
<eviction strategy="LIRS" maxEntries="600"/>
<!-- Configure a synchronous replication cache -->
<clustering mode="replication">
<sync/>
</clustering>
<transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup" transactionMode="TRANSACTIONAL" lockingMode="PESSIMISTIC"/>
<loaders passivation="false" shared="false" preload="false">
<loader class="org.infinispan.loaders.jdbm.JdbmCacheStore" fetchPersistentState="true" purgeOnStartup="false">
<properties>
<property name="location" value="D:\test\repository\longterm\content"/>
</properties>
</loader>
</loaders>
</namedCache>
</infinispan>
ModeShape configuration:
{
"name" : "longterm",
"monitoring" : {
"enabled" : false
},
"workspaces" : {
"predefined" : [ "otherWorkspace" ],
"default" : "default",
"allowCreation" : true
},
"security" : {
"anonymous" : {
"roles" : [ "readonly", "readwrite", "admin" ],
"useOnFailedLogin" : false
}
},
"storage" : {
"cacheConfiguration" : "D:\\test\\conf\\longterm-infinispan-configuration.xml",
"cacheName" : "persisted-repository",
"binaryStorage" : {
"type" : "file",
"directory" : "D:\\test\\repository\\longterm\\binaries",
"minimumBinarySizeInBytes" : 999
}
},
"query" : {
"enabled" : true,
"indexStorage" : {
"type" : "filesystem",
"location" : "D:\\test\\repository\\longterm\\indexes",
"lockingStrategy" : "simple",
"fileSystemAccessType" : "auto"
},
"textExtracting" : {
"extractors" : {
"tikaExtractor" : {
"name" : "General content-based extractor",
"classname" : "tika"
}
}
},
"indexing" : {
"rebuildOnStartup" : {
"when" : "if_missing",
"mode" : "async"
}
}
},
"clustering" : {
"clusterName" : "my-repo-cluster",
"channelConfiguration" : "jgroups-config.xml"
}
}
If I remove the configuration to use the "jgroups-config.xml" from both of these configuration files so that it will use the jgroups default configuration then I no longer get the errors logged in the log files, but I now see the following warning in the stderr log which is constantly repeated.
Dec 30, 2014 12:11:14 PM org.jgroups.logging.JDKLogImpl warn
WARNING: SVP321APP-53335: no physical address for 867214b2-c0af-a43e-4792-9e748f12801f, dropping message
Running in the default jgroups configuraiton I can use wireshark and see that messages are sent from each instance to the other instance so it appears there is some communication between the two instances. But when I add something to one instance of modeshape it never gets replicated.
As an added test I simply ran the jgroups draw demo on each of the servers as described here: http://www.jgroups.org/tutorial/index.html. Rather than running two instances on the same server, I ran one instance on each server and as I draw on one I can see the results on the other server. So I know that jgroups in its default configuration is working as far as this demo app is concerned.
The frustrating thing is that whatever I have tried has not worked to cause any replication to occur. It is almost as if ModeShape is not even attempting to do any replication. I see nothing in the logs that it is even attempting to do anything. All I see is the wireshark communication between the instances of JGroups, but in none of that communication do I see anything regarding replicating data.
At this point I don't have any idea on what to try or how to debug the problem of no replication. I would appreciate if anyone can help me understand what I am missing, or where I can get further info as to what may be going wrong, or how to debug this problem further.
Thanks.
