2 Replies Latest reply on Dec 19, 2012 6:33 PM by pferraro

    Problems using JGroups TUNNEL on JBoss AS7 7.2.0-Alpha

    pillingworth

      I am trying to use JGroups TUNNEL protocol for JBoss AS7 (7.2.0-Alpha1 built from ed2bc551a55ec6a8167a8657cbb5d8abc6e07748).

       

      My config is as follows

       

      <stack name="tunnel">

          <transport type="TUNNEL" shared="false">

              <property name="enable_bundling">false</property>

              <property name="gossip_router_hosts">SALAK.saa-cons.co.uk[12000]</property>

          </transport>

          <protocol type="PING"/>

          <protocol type="MERGE2"/>

          <protocol type="FD"/>

          <protocol type="VERIFY_SUSPECT"/>

          <protocol type="BARRIER"/>

          <protocol type="pbcast.NAKACK"/>

          <protocol type="UNICAST2"/>

          <protocol type="pbcast.STABLE"/>

          <protocol type="pbcast.GMS"/>

          <protocol type="UFC"/>

          <protocol type="MFC"/>

          <protocol type="FRAG2"/>

      </stack>

       

      I am running the Gossip Router and all the nodes on the same physical machine.

       

      I am starting a GossipRouter using

       

      java -cp modules\org\jgroups\main\jgroups-3.2.0.Final.jar org.jgroups.stack.GossipRouter -port 12000

       

      When I start the first node in my cluster it seems to start ok. It connects to the Gossip Router (can see by dumping details using the Infinispan MBeans) and there are no errors logged.

       

      When I start the second node I get errors straight away, logged on both nodes. No errors are reported by the Gossip Router itself. Both nodes are connected to the Gossip Router (again, from dumping details using the Infinispan MBeans).

       

      Is this a bug, known issue or a problem with my configuration?

       

      Exception from the first node

      2012-12-10 10:19:34,903 ERROR [org.infinispan.interceptors.InvocationContextInterceptor](pool-9-thread-1) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to onmatte-salak/DOMAIN.PAULI.V70

          at org.infinispan.util.Util.rewrapAsCacheException(Util.java:532)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:185)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:177)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:199)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:256)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:243)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:238)

          at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:217)

          at org.infinispan.interceptors.ReplicationInterceptor.broadcastPrepare(ReplicationInterceptor.java:133)

          at org.infinispan.interceptors.ReplicationInterceptor.visitPrepareCommand(ReplicationInterceptor.java:125)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:94)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.invokeNextAndCommitIf1Pc(AbstractTxLockingInterceptor.java:110)

          at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:135)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.TxInterceptor.invokeNextInterceptorAndVerifyTransaction(TxInterceptor.java:131)

          at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:118)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:209)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleTxCommand(StateTransferInterceptor.java:188)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitPrepareCommand(StateTransferInterceptor.java:92)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

          at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:85)

          at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:126)

          at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:133)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)

          at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:141)

          at org.infinispan.transaction.TransactionCoordinator.prepare(TransactionCoordinator.java:123)

          at org.infinispan.transaction.synchronization.SynchronizationAdapter.beforeCompletion(SynchronizationAdapter.java:68)

          at org.infinispan.transaction.tm.DummyTransaction.notifyBeforeCompletion(DummyTransaction.java:230)

          at org.infinispan.transaction.tm.DummyTransaction.runPrepare(DummyTransaction.java:242)

          at org.infinispan.transaction.tm.DummyTransaction.commit(DummyTransaction.java:86)

          at org.infinispan.transaction.tm.DummyBaseTransactionManager.commit(DummyBaseTransactionManager.java:102)

          at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1017)

          at org.infinispan.CacheImpl.put(CacheImpl.java:703)

          at org.infinispan.CacheImpl.put(CacheImpl.java:695)

          at org.infinispan.CacheSupport.put(CacheSupport.java:128)

          at org.infinispan.AbstractDelegatingCache.put(AbstractDelegatingCache.java:100)

          at com.saaconsultants.management.registry.infinispan.svc.InfinispanRegistryHandlerSvc.put(InfinispanRegistryHandlerSvc.java:552)

          at com.saaconsultants.management.registry.infinispan.svc.InfinispanRegistryHandlerSvc.put(InfinispanRegistryHandlerSvc.java:164)

          at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318)

          at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)

          at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

          at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)

          at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)

          at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)

          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

          at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)

          at $Proxy52.put(Unknown Source)

          at com.saaconsultants.management.agent.adverts.MultiResourceAdvertiser.updateAdvert(MultiResourceAdvertiser.java:199)

          at com.saaconsultants.management.agent.adverts.MultiResourceAdvertiser.updateAdverts(MultiResourceAdvertiser.java:164)

          at com.saaconsultants.management.agent.adverts.MultiResourceAdvertiser.readvertise(MultiResourceAdvertiser.java:156)

          at com.saaconsultants.management.agent.adverts.MultiResourceAdvertiser.access$000(MultiResourceAdvertiser.java:26)

          at com.saaconsultants.management.agent.adverts.MultiResourceAdvertiser$1.run(MultiResourceAdvertiser.java:90)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

          at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)

          at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)

          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

      Caused by: org.jgroups.TimeoutException: timeout sending message to onmatte-salak/DOMAIN.PAULI.V70

          at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:299)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:178)

          ... 91 more

       

      Exception from the second node

      2012-12-10 10:20:12,278 ERROR [org.jboss.msc.service.fail](ServerService Thread Pool -- 50) MSC00001: Failed to start service jboss.infinispan."DOMAIN.PAULI.V70".remote-service-map: org.jboss.msc.service.StartException in service jboss.infinispan."DOMAIN.PAULI.V70".remote-service-map: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl

          at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

          at org.jboss.threads.JBossThread.run(JBossThread.java:122)

      Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl

          at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)

          at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)

          at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)

          at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)

          at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)

          at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)

          at org.infinispan.CacheImpl.start(CacheImpl.java:520)

          at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)

          at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)

          at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)

          at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)

          at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)

          at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)

          at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)

          at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)

          ... 4 more

      Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache remote-service-map on onmatte-salak/DOMAIN.PAULI.V70

          at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)

          ... 18 more

       

      2012-12-10 10:20:12,278 ERROR [org.jboss.msc.service.fail](ServerService Thread Pool -- 49) MSC00001: Failed to start service jboss.infinispan."DOMAIN.PAULI.V70".reims-registry: org.jboss.msc.service.StartException in service jboss.infinispan."DOMAIN.PAULI.V70".reims-registry: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl

          at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:87)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)

          at org.jboss.threads.JBossThread.run(JBossThread.java:122)

      Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl

          at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)

          at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:883)

          at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:654)

          at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:643)

          at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:546)

          at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:199)

          at org.infinispan.CacheImpl.start(CacheImpl.java:520)

          at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:690)

          at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:653)

          at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:549)

          at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:563)

          at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:107)

          at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:98)

          at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)

          at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:82)

          ... 4 more

      Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache reims-registry on onmatte-salak/DOMAIN.PAULI.V70

          at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:209)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)

          ... 18 more