0 Replies Latest reply on Nov 5, 2010 6:16 PM by fokker

    Failed to replicate session / failed unmarshalling message

    fokker

      Hello everyone, we're currently experiencing a problem when trying to replicate sessions via UDP multicast (it works just fine over TCP). We're currently using JBoss 4.2.3.GA

       

      This cluster has only two machines: If you log in on machine A, the session should be automatically replicated to server B. The problem occurs when you access the site on any of the two servers.

       

      On the server getting "accessed" i'm getting:

       

      22:05:01,684 WARN  [/atmosphir] Failed to replicate session 3Q2eCDboj1FVZSgfk-ri1w**.node-10-204-238-6
      java.lang.RuntimeException: JBossCacheService: exception occurred in cache put ...
      at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:147)
      at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:325)
      at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:123)
      at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1127)
      at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:682)
      at org.jboss.web.tomcat.service.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)
      at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:108)
      at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84)
      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
      at java.lang.Thread.run(Thread.java:619)
      Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.31.1.1:56138, retval=null, received=false, suspected=false
      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4449)
      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4371)
      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4489)
      at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)
      at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)
      at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:124)
      at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:379)
      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:174)
      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)
      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)
      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5919)
      at org.jboss.cache.TreeCache.put(TreeCache.java:3784)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy104.put(Unknown Source)
      at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:138)
      ... 17 more
      Caused by: org.jboss.cache.lock.TimeoutException: Response timed out: sender=172.31.1.1:56138, retval=null, received=false, suspected=false
      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4447)
      ... 42 more

      22:05:01,684 WARN  [/test] Failed to replicate session 3Q2eCDboj1FVZSgfk-ri1w**.node-10-204-238-6

      java.lang.RuntimeException: JBossCacheService: exception occurred in cache put ...

      at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:147)

      at org.jboss.web.tomcat.service.session.JBossCacheService.putSession(JBossCacheService.java:325)

      at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.processSessionRepl(JBossCacheClusteredSession.java:123)

      at org.jboss.web.tomcat.service.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:1127)

      at org.jboss.web.tomcat.service.session.JBossCacheManager.storeSession(JBossCacheManager.java:682)

      at org.jboss.web.tomcat.service.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:49)

      at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:108)

      at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:84)

      at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)

      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

      at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)

      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)

      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)

      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)

      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)

      at java.lang.Thread.run(Thread.java:619)

      Caused by: org.jboss.cache.ReplicationException: rsp=sender=172.31.1.1:56138, retval=null, received=false, suspected=false

      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4449)

      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4371)

      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4489)

      at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:110)

      at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:88)

      at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:124)

      at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88)

      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)

      at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:379)

      at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:174)

      at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:68)

      at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:167)

      at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:5919)

      at org.jboss.cache.TreeCache.put(TreeCache.java:3784)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)

      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)

      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)

      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)

      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)

      at $Proxy104.put(Unknown Source)

      at org.jboss.web.tomcat.service.session.JBossCacheWrapper.put(JBossCacheWrapper.java:138)

      ... 17 more

      Caused by: org.jboss.cache.lock.TimeoutException: Response timed out: sender=172.31.1.1:56138, retval=null, received=false, suspected=false

      at org.jboss.cache.TreeCache.callRemoteMethods(TreeCache.java:4447)

      ... 42 more


      And on the other node i'm getting:

       

       

      22:05:01,810 ERROR [UDP] failed unmarshalling message

      java.io.EOFException

      at java.io.DataInputStream.readShort(DataInputStream.java:298)

      at org.jgroups.Message.readFrom(Message.java:630)

      at org.jgroups.protocols.TP.bufferToList(TP.java:1025)

      at org.jgroups.protocols.TP.handleIncomingPacket(TP.java:828)

      at org.jgroups.protocols.TP.access$400(TP.java:46)

      at org.jgroups.protocols.TP$IncomingPacketHandler.run(TP.java:1297)

      at java.lang.Thread.run(Thread.java:619)


      Needless to say, both servers can ping and "auto discover" each other just fine, but they are being unable to replicate sessions for some unknown reason.
      I'm starting both servers using the following parameters:
      Server A:
      run.sh -c all -g MyCluster -u 239.255.101.100 -b 172.31.1.1

      Server B:
      run.sh -c all -g MyCluster -u 239.255.101.100 -b 172.31.1.5

      Does anyone have any idea of what might be going on?