Periodic org.jboss.aop.NotFoundInDispatcherException prevent
We are running JBoss Messaging 1.0.1SP4. After some period of use (seems to happen after many messages have been sent through the system, but not necessarily related to load), we see the messages below in the logs, and all further sends seem to fail until the server is restarted. Once the server is restarted, the client automatically reconnects, and everything works again.
Server Log of org.jboss.messaging and org.jboss.remoting at debug level:
Removed org.jboss.remoting.transport.local.LocalClientInvoker@686ecb from registry Using local client invoker for invocation. ServerInvoker (SocketServerInvoker[172.31.40.54:2195]) removing client callback handler with session id of 4sv141i-vyg6ek-eyta2mrp-1-ez07tnm4-2wh+4sv141i-vyg6ek-eyta2mrp-1-ez07tnm4-2wi. removed org.jboss.remoting.transport.local.LocalClientInvoker@4201d9 from registry decremented org.jboss.remoting.transport.local.LocalClientInvoker@e6c243's count, current count 12 Server connection lost to client (session id = 4sv141k-a5vaz8-ez00omj4-1-ez00oo3z-4 SocketServerInvoker[0.0.0.0:4457].invoke() call failed: Object with oid: -2147481983 was not found in the Dispatcher SocketServerInvoker[0.0.0.0:4457].invoke() call failed org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147481983 was not found in the Dispatcher at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) SocketServerInvoker[0.0.0.0:4457].invoke() call failed: Object with oid: -2147481983 was not found in the Dispatcher SocketServerInvoker[0.0.0.0:4457].invoke() call failed org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147481983 was not found in the Dispatcher at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) ...
Client Log of org.jboss.messaging at debug level:
describeObject for class=org.jboss.remoting.InvocationRequest describeObject::a new reference 1 defaultWrite::org.jboss.remoting.InvocationRequest contains 1 slots defaultWrite:: slot 0 NR=org.jboss.remoting.InvocationRequest from parentClass=org.jboss.remoting.InvocationRequest writeSlotWithFields slot=org.jboss.remoting.InvocationRequest and 6 fields writeSlotWithFields FieldNr=0 writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField sessionId with object=java.lang.String writeSlotWithFields FieldNr=1 writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField subsystem with object=java.lang.String writeSlotWithFields FieldNr=2 writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField arg with object=org.jboss.jms.server.remoting.MessagingMarshallable describeObject for class=org.jboss.jms.server.remoting.MessagingMarshallable describeObject::a new reference 5 defaultWrite::org.jboss.jms.server.remoting.MessagingMarshallable contains 1 slots defaultWrite:: slot 0 NR=org.jboss.jms.server.remoting.MessagingMarshallable from parentClass=org.jboss.jms.server.remoting.MessagingMarshallable writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable and 2 fields writeSlotWithFields FieldNr=0 writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable primitiveField version with object=NULL writeSlotWithFields FieldNr=1 writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable objectField load with object=org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442 describeObject for class=org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442 describeObject::a new reference 7 describeObject for class=org.jboss.aop.metadata.SimpleMetaData describeObject::a new reference 9 writeSlotWithFields FieldNr=3 writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField requestPayload with object=NULL writeSlotWithFields FieldNr=4 writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField returnPayload with object=NULL writeSlotWithFields FieldNr=5 writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField locator with object=NULL objectFromDescription::reading new definition Reading string with utfSize=37 isLong=false readString::pulling data to Buffer at pos 0 size= 0 hash on field org.jboss.remoting.InvocationResponse = -5123635159962572581 lookupMethodOnHierarchy::class=org.jboss.remoting.InvocationResponse looking for readResolve lookupMethodOnHierarchy::currentClass=class org.jboss.remoting.InvocationResponse lookupMethodOnHierarchy::class=org.jboss.remoting.InvocationResponse looking for writeReplace lookupMethodOnHierarchy::currentClass=class org.jboss.remoting.InvocationResponse hash on field org.jboss.remoting.InvocationResponse = -5123635159962572581 hash on field java.lang.String$sessionId = 7208879560699038148 hash on field boolean$isException = 6944741943427072418 hash on field java.lang.Object$result = -1910178885101376583 hash on field java.util.Map$payload = -311608580347357596 Reading object for id=1 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = org.jboss.remoting.InvocationResponse defaultRead::class org.jboss.remoting.InvocationResponse contains 1 slots defaultRead::slot[0]=org.jboss.remoting.InvocationResponse readSlotWithFields slot=org.jboss.remoting.InvocationResponse FieldName on Read=sessionId readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=sessionId<<-reading Object Reading string with utfSize=36 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=isException FieldName on Read=result readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=result<<-reading Object objectFromDescription::reading new definition Reading string with utfSize=43 isLong=false readString::pulling data to Buffer at pos 0 size= 0 hash on field org.jboss.aop.NotFoundInDispatcherException = -5393238603831683671 lookupMethodOnHierarchy::class=org.jboss.aop.NotFoundInDispatcherException looking for readResolve lookupMethodOnHierarchy::currentClass=class org.jboss.aop.NotFoundInDispatcherException lookupMethodOnHierarchy::currentClass=class org.jboss.aop.DispatcherConnectException lookupMethodOnHierarchy::currentClass=class java.lang.Exception lookupMethodOnHierarchy::currentClass=class java.lang.Throwable lookupMethodOnHierarchy::class=org.jboss.aop.NotFoundInDispatcherException looking for writeReplace lookupMethodOnHierarchy::currentClass=class org.jboss.aop.NotFoundInDispatcherException lookupMethodOnHierarchy::currentClass=class org.jboss.aop.DispatcherConnectException lookupMethodOnHierarchy::currentClass=class java.lang.Exception lookupMethodOnHierarchy::currentClass=class java.lang.Throwable hash on field org.jboss.aop.NotFoundInDispatcherException = -5393238603831683671 hash on field java.lang.Object$oid = 5891181997928938037 hash on field org.jboss.aop.DispatcherConnectException = -2548359991833714121 hash on field java.lang.Exception = 7600092213321646919 hash on field java.lang.Throwable = -7779525601374451073 hash on field java.lang.String$detailMessage = -4333528023742127755 hash on field java.lang.Throwable$cause = 4080765070623846565 hash on field [Ljava.lang.StackTraceElement;$stackTrace = -4842200784206624673 Reading object for id=4 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = org.jboss.aop.NotFoundInDispatcherException defaultRead::class org.jboss.aop.NotFoundInDispatcherException contains 4 slots defaultRead::slot[0]=java.lang.Throwable readSlotWithFields slot=java.lang.Throwable FieldName on Read=detailMessage readSlotWithFields slot=java.lang.Throwable field=detailMessage<<-reading Object Reading string with utfSize=60 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=cause readSlotWithFields slot=java.lang.Throwable field=cause<<-reading Object objectFromDescription::reading circular definition reference=4 FieldName on Read=stackTrace readSlotWithFields slot=java.lang.Throwable field=stackTrace<<-reading Object objectFromDescription::reading new definition Reading string with utfSize=30 isLong=false readString::pulling data to Buffer at pos 0 size= 0 hash on field [Ljava.lang.StackTraceElement; = -8287581384409355425 lookupMethodOnHierarchy::class=[Ljava.lang.StackTraceElement; looking for readResolve lookupMethodOnHierarchy::currentClass=class [Ljava.lang.StackTraceElement; lookupMethodOnHierarchy::class=[Ljava.lang.StackTraceElement; looking for writeReplace lookupMethodOnHierarchy::currentClass=class [Ljava.lang.StackTraceElement; hash on field [Ljava.lang.StackTraceElement; = -8287581384409355425 Reading object for id=7 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = [Ljava.lang.StackTraceElement; objectFromDescription::reading new definition Reading string with utfSize=27 isLong=false readString::pulling data to Buffer at pos 0 size= 0 hash on field java.lang.StackTraceElement = -5341604412408429699 lookupMethodOnHierarchy::class=java.lang.StackTraceElement looking for readResolve lookupMethodOnHierarchy::currentClass=class java.lang.StackTraceElement lookupMethodOnHierarchy::class=java.lang.StackTraceElement looking for writeReplace lookupMethodOnHierarchy::currentClass=class java.lang.StackTraceElement hash on field java.lang.StackTraceElement = -5341604412408429699 hash on field java.lang.String$declaringClass = -4020810388859008069 hash on field java.lang.String$methodName = 2043831539205426629 hash on field java.lang.String$fileName = -3886410644765885867 hash on field int$lineNumber = 1246520928412557129 Reading object for id=9 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement defaultRead::class java.lang.StackTraceElement contains 1 slots defaultRead::slot[0]=java.lang.StackTraceElement readSlotWithFields slot=java.lang.StackTraceElement FieldName on Read=declaringClass readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object Reading string with utfSize=24 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=methodName readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object Reading string with utfSize=6 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=fileName readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object Reading string with utfSize=15 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=lineNumber objectFromDescription::reading new definition Reading object for id=14 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement defaultRead::class java.lang.StackTraceElement contains 1 slots defaultRead::slot[0]=java.lang.StackTraceElement readSlotWithFields slot=java.lang.StackTraceElement FieldName on Read=declaringClass readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object Reading string with utfSize=56 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=methodName readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object FieldName on Read=fileName readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object Reading string with utfSize=31 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=lineNumber objectFromDescription::reading new definition Reading object for id=17 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement defaultRead::class java.lang.StackTraceElement contains 1 slots defaultRead::slot[0]=java.lang.StackTraceElement readSlotWithFields slot=java.lang.StackTraceElement FieldName on Read=declaringClass readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object Reading string with utfSize=32 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=methodName readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object FieldName on Read=fileName readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object Reading string with utfSize=18 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=lineNumber objectFromDescription::reading new definition Reading object for id=20 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement defaultRead::class java.lang.StackTraceElement contains 1 slots defaultRead::slot[0]=java.lang.StackTraceElement readSlotWithFields slot=java.lang.StackTraceElement FieldName on Read=declaringClass readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object Reading string with utfSize=48 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=methodName readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object Reading string with utfSize=17 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=fileName readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object Reading string with utfSize=17 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=lineNumber objectFromDescription::reading new definition Reading object for id=24 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement defaultRead::class java.lang.StackTraceElement contains 1 slots defaultRead::slot[0]=java.lang.StackTraceElement readSlotWithFields slot=java.lang.StackTraceElement FieldName on Read=declaringClass readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object FieldName on Read=methodName readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object Reading string with utfSize=5 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=fileName readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object FieldName on Read=lineNumber objectFromDescription::reading new definition Reading object for id=26 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement defaultRead::class java.lang.StackTraceElement contains 1 slots defaultRead::slot[0]=java.lang.StackTraceElement readSlotWithFields slot=java.lang.StackTraceElement FieldName on Read=declaringClass readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object FieldName on Read=methodName readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object Reading string with utfSize=3 isLong=false readString::pulling data to Buffer at pos 0 size= 0 FieldName on Read=fileName readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object FieldName on Read=lineNumber defaultRead::slot[1]=java.lang.Exception readSlotWithFields slot=java.lang.Exception defaultRead::slot[2]=org.jboss.aop.DispatcherConnectException readSlotWithFields slot=org.jboss.aop.DispatcherConnectException defaultRead::slot[3]=org.jboss.aop.NotFoundInDispatcherException readSlotWithFields slot=org.jboss.aop.NotFoundInDispatcherException FieldName on Read=oid readSlotWithFields slot=org.jboss.aop.NotFoundInDispatcherException field=oid<<-reading Object FieldName on Read=payload readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=payload<<-reading Object Caught Exception: org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483579 was not found in the Dispatcher at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163) at org.jboss.remoting.Client.invoke(Client.java:1414) at org.jboss.remoting.Client.invoke(Client.java:511) at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107) at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java) at org.jboss.jms.client.delegate.ClientConnectionDelegate.stop(ClientConnectionDelegate.java) at org.jboss.jms.client.JBossConnection.stop(JBossConnection.java:126) at oracle.tip.adapter.jms.JMS.JMSConnection.stop(JMSConnection.java:432) at oracle.tip.adapter.jms.JMS.JMSMessageConsumer.stop(JMSMessageConsumer.java:252) at oracle.tip.adapter.jms.inbound.JmsConsumer.stop(JmsConsumer.java:164) at oracle.tip.adapter.jms.JmsEndpoint.stop(JmsEndpoint.java:150) at oracle.tip.adapter.jms.JmsResourceAdapter.endpointDeactivation(JmsResourceAdapter.java:196) at oracle.tip.adapter.fw.jca.AdapterFrameworkImpl.endpointDeactivation(AdapterFrameworkImpl.java:590) at oracle.tip.adapter.fw.agent.jca.JCAActivationAgent.deactivateInboundJcaEndpoint(JCAActivationAgent.java:953) at oracle.tip.adapter.fw.agent.jca.JCAActivationAgent.uninit(JCAActivationAgent.java:458) at com.collaxa.cube.engine.core.BaseCubeProcess.disableActivationAgents(BaseCubeProcess.java:516) at com.collaxa.cube.engine.core.BaseCubeProcess.uninit(BaseCubeProcess.java:321) at com.collaxa.cube.engine.deployment.CubeProcessLoader.destroy(CubeProcessLoader.java:589) at com.collaxa.cube.engine.deployment.CubeProcessHolder.uninit(CubeProcessHolder.java:141) at com.collaxa.cube.engine.CubeEngine.uninit(CubeEngine.java:794) at com.collaxa.cube.admin.ServerManager.uninit(ServerManager.java:334) at com.collaxa.cube.admin.ServerManager$1.run(ServerManager.java:150)
Any ideas on where to go from here to get some possible idea what may be causing this error?