What does this ReplicationException mean?

[jamieqho]

Hi,

I just realized that my application should be using the cache in REPL_SYNC mode, not REPL_ASYNC. The caches have to always be in sync.

Now I notice that my servers are throwing ReplicationExceptions. They seem to be colliding and deadlocking. Is this a fatal error? Should my servers be shutting down when this happens or is it okay to just print the error and proceed?

Thanks,
Jamie


Server 1:

org.jboss.cache.ReplicationException: rsp=sender=10.133.192.106:3439, retval=null, received=false, suspected=false
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1984)
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1898)
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:2009)
at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:72)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:91)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:63)
at

Server 2:

org.jboss.cache.ReplicationException: rsp=sender=10.133.192.106:3348, retval=nul
l, received=false, suspected=false
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1984)
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1898)
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:2009)
at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:
72)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpc
Interceptor.java:91)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpc
Interceptor.java:63)
at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedM
ethod(ReplicationInterceptor.java:116)
at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(Replicatio
nInterceptor.java:88)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:75)
at org.jboss.cache.interceptors.NotificationInterceptor.invoke(Notificat

[jamieqho]

Here is the entire stack trace in case you need it:

org.jboss.cache.ReplicationException: rsp=sender=10.133.192.106:3439, retval=null, received=false, suspected=false
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1984)
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1898)
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:2009)
at org.jboss.cache.RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:72)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:91)
at org.jboss.cache.interceptors.BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:63)
at org.jboss.cache.interceptors.ReplicationInterceptor.handleReplicatedMethod(ReplicationInterceptor.java:116)
at org.jboss.cache.interceptors.ReplicationInterceptor.invoke(ReplicationInterceptor.java:88)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:75)
at org.jboss.cache.interceptors.NotificationInterceptor.invoke(NotificationInterceptor.java:22)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:75)
at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:294)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:128)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:75)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:142)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:75)
at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:66)
at org.jboss.cache.CacheImpl.invokeMethod(CacheImpl.java:3752)
at org.jboss.cache.CacheImpl.put(CacheImpl.java:1480)
at org.jboss.cache.UnversionedNode.put(UnversionedNode.java:214)
at org.jboss.cache.UnversionedNode.putIfAbsent(UnversionedNode.java:445)
at com.adobe.acorn.util.ExpirableCache.putIfAbsent(ExpirableCache.java:74)
at com.adobe.acorn.meeting.MeetingManager.getMeetingSession(MeetingManager.java:110)
at com.adobe.acorn.ws.SystemServlet.doGet(SystemServlet.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:498)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:367)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:185)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:715)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:401)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:458)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:776)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:500)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:209)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:358)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:336)
at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
Caused by: org.jboss.cache.lock.TimeoutException: Replication timeout for 10.133.192.106:3439
at org.jboss.cache.CacheImpl.callRemoteMethods(CacheImpl.java:1982)
... 42 more

[jamieqho]

Sorry, one more thing... those IPs I gave above are the same because I am running two servers on the same machine. Would that cause replication time outs like this?

Thanks,
Jamie

[manik]

Hmm, you need to look at why nodes drop out. Perhaps your sync repl timeouts are too low? What version of JGroups are you using, and what is your cluster config section of your cache config like?