JBoss EAP5.2 clusterred nodes are giving error : java.lang.IllegalStateException: Don't know about node id:3..... and java.lang.IllegalStateException: Failed to remove binding, and..... java.lang.IllegalStateException: Cannot find object in dispatcher
Hi All,
I am getting below error in production env. I am using 4 nodes and EAP-5.2
Error1:
Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a
java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a
======================================================================================================
org.jboss.jms.exception.MessagingJMSException: Failed to invoke
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$start$aop(ClientConnectionDelegate.java:264)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeTarget(ClientConnectionDelegate$start_N8025343665958530775.java)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ConnectionAspect.handleStart(ConnectionAspect.java:147)
2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source)
2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2015-06-01 08:52:08,785 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:703)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:758)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:244)
Error2
=======================
ERROR [org.jboss.messaging.core.impl.postoffice.UnbindRequest] Failed to remove binding
java.lang.IllegalStateException: Don't know about node id: 3
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBindingFromCluster(MessagingPostOffice.java:1912)
at org.jboss.messaging.core.impl.postoffice.UnbindRequest.execute(UnbindRequest.java:61)
at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:725)
at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:616)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
at org.jgroups.JChannel.up(JChannel.java:1336)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:494)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
at org.jgroups.protocols.FC.up(FC.java:473)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)
at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:176)
at org.jgroups.protocols.FD.up(FD.java:284)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:264)
at org.jgroups.protocols.PING.up(PING.java:273)
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
at org.jgroups.protocols.TP.access$100(TP.java:50)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
2015-06-01 04:27:02,017 ERROR [org.jboss.messaging.core.impl.postoffice.GroupMember] Caught Exception in RequestHandler
java.lang.Exception: Failed to remove binding
at org.jboss.messaging.core.impl.postoffice.UnbindRequest.execute(UnbindRequest.java:67)
at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:725)
at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:616)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:637)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:545)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:368)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:775)
at org.jgroups.JChannel.up(JChannel.java:1336)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:454)
at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:494)
at org.jgroups.protocols.pbcast.STATE_TRANSFER.up(STATE_TRANSFER.java:153)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:188)
at org.jgroups.protocols.FC.up(FC.java:473)
at org.jgroups.protocols.pbcast.GMS.up(GMS.java:821)
at org.jgroups.protocols.VIEW_SYNC.up(VIEW_SYNC.java:192)
at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:233)
at org.jgroups.protocols.UNICAST.up(UNICAST.java:328)
at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:895)
at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:708)
at org.jgroups.protocols.BARRIER.up(BARRIER.java:136)
at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:176)
at org.jgroups.protocols.FD.up(FD.java:284)
at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:328)
at org.jgroups.protocols.MERGE2.up(MERGE2.java:144)
at org.jgroups.protocols.Discovery.up(Discovery.java:264)
at org.jgroups.protocols.PING.up(PING.java:273)
at org.jgroups.protocols.TP$ProtocolAdapter.up(TP.java:2319)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1253)
at org.jgroups.protocols.TP.access$100(TP.java:50)
at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1830)
at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1809)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.IllegalStateException: Don't know about node id: 3
Error3:
==================
org.jboss.jms.exception.MessagingJMSException: Failed to invoke
==============================================================
2015-06-01 08:51:52,940 INFO [STDOUT] ********* E N D O F E R R O R R E P O R T *********
2015-06-01 08:51:54,035 ERROR [STDERR] org.jboss.jms.exception.MessagingJMSException: Failed to invoke
2015-06-01 08:51:54,036 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
2015-06-01 08:51:54,036 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
2015-06-01 08:51:54,036 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$start$aop(ClientConnectionDelegate.java:264)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeTarget(ClientConnectionDelegate$start_N8025343665958530775.java)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.container.ConnectionAspect.handleStart(ConnectionAspect.java:147)
2015-06-01 08:51:54,037 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source)
2015-06-01 08:51:54,037 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2015-06-01 08:51:54,037 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:703)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:758)
2015-06-01 08:51:54,037 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:244)
Error4:
==================
Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a
java.lang.IllegalStateException: Cannot find object in dispatcher with id ttb8-vdc0tdai-1-gkuzs3ai-p0mxki-al481a
======================================================================================================
org.jboss.jms.exception.MessagingJMSException: Failed to invoke
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$start$aop(ClientConnectionDelegate.java:264)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate$start_N8025343665958530775.invokeTarget(ClientConnectionDelegate$start_N8025343665958530775.java)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ConnectionAspect.handleStart(ConnectionAspect.java:147)
2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.GeneratedMethodAccessor388.invoke(Unknown Source)
2015-06-01 08:52:08,785 ERROR [STDERR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2015-06-01 08:52:08,785 ERROR [STDERR] at java.lang.reflect.Method.invoke(Method.java:597)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.delegate.ClientConnectionDelegate.start(ClientConnectionDelegate.java)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.jms.client.JBossConnection.start(JBossConnection.java:121)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsManagedConnection.start(JmsManagedConnection.java:703)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSession.start(JmsSession.java:758)
2015-06-01 08:52:08,785 ERROR [STDERR] at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.start(JmsSessionFactoryImpl.java:244)