0 Replies Latest reply on Mar 7, 2007 5:21 PM by Mike Poindexter

    Periodic org.jboss.aop.NotFoundInDispatcherException prevent

    Mike Poindexter Newbie

      We are running JBoss Messaging 1.0.1SP4. After some period of use (seems to happen after many messages have been sent through the system, but not necessarily related to load), we see the messages below in the logs, and all further sends seem to fail until the server is restarted. Once the server is restarted, the client automatically reconnects, and everything works again.

      Server Log of org.jboss.messaging and org.jboss.remoting at debug level:

      Removed org.jboss.remoting.transport.local.LocalClientInvoker@686ecb from registry
      Using local client invoker for invocation.
      ServerInvoker (SocketServerInvoker[172.31.40.54:2195]) removing client callback handler with session id of 4sv141i-vyg6ek-eyta2mrp-1-ez07tnm4-2wh+4sv141i-vyg6ek-eyta2mrp-1-ez07tnm4-2wi.
      removed org.jboss.remoting.transport.local.LocalClientInvoker@4201d9 from registry
      decremented org.jboss.remoting.transport.local.LocalClientInvoker@e6c243's count, current count 12
      Server connection lost to client (session id = 4sv141k-a5vaz8-ez00omj4-1-ez00oo3z-4
      SocketServerInvoker[0.0.0.0:4457].invoke() call failed: Object with oid: -2147481983 was not found in the Dispatcher
      SocketServerInvoker[0.0.0.0:4457].invoke() call failed
      org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147481983 was not found in the Dispatcher
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      SocketServerInvoker[0.0.0.0:4457].invoke() call failed: Object with oid: -2147481983 was not found in the Dispatcher
      SocketServerInvoker[0.0.0.0:4457].invoke() call failed
      org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147481983 was not found in the Dispatcher
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
      ...
      


      Client Log of org.jboss.messaging at debug level:
      describeObject for class=org.jboss.remoting.InvocationRequest
      describeObject::a new reference 1
      defaultWrite::org.jboss.remoting.InvocationRequest contains 1 slots
      defaultWrite:: slot 0 NR=org.jboss.remoting.InvocationRequest from parentClass=org.jboss.remoting.InvocationRequest
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest and 6 fields
      writeSlotWithFields FieldNr=0
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField sessionId with object=java.lang.String
      writeSlotWithFields FieldNr=1
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField subsystem with object=java.lang.String
      writeSlotWithFields FieldNr=2
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField arg with object=org.jboss.jms.server.remoting.MessagingMarshallable
      describeObject for class=org.jboss.jms.server.remoting.MessagingMarshallable
      describeObject::a new reference 5
      defaultWrite::org.jboss.jms.server.remoting.MessagingMarshallable contains 1 slots
      defaultWrite:: slot 0 NR=org.jboss.jms.server.remoting.MessagingMarshallable from parentClass=org.jboss.jms.server.remoting.MessagingMarshallable
      writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable and 2 fields
      writeSlotWithFields FieldNr=0
      writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable primitiveField version with object=NULL
      writeSlotWithFields FieldNr=1
      writeSlotWithFields slot=org.jboss.jms.server.remoting.MessagingMarshallable objectField load with object=org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442
      describeObject for class=org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442
      describeObject::a new reference 7
      describeObject for class=org.jboss.aop.metadata.SimpleMetaData
      describeObject::a new reference 9
      writeSlotWithFields FieldNr=3
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField requestPayload with object=NULL
      writeSlotWithFields FieldNr=4
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField returnPayload with object=NULL
      writeSlotWithFields FieldNr=5
      writeSlotWithFields slot=org.jboss.remoting.InvocationRequest objectField locator with object=NULL
      objectFromDescription::reading new definition
      Reading string with utfSize=37 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      hash on field org.jboss.remoting.InvocationResponse = -5123635159962572581
      lookupMethodOnHierarchy::class=org.jboss.remoting.InvocationResponse looking for readResolve
      lookupMethodOnHierarchy::currentClass=class org.jboss.remoting.InvocationResponse
      lookupMethodOnHierarchy::class=org.jboss.remoting.InvocationResponse looking for writeReplace
      lookupMethodOnHierarchy::currentClass=class org.jboss.remoting.InvocationResponse
      hash on field org.jboss.remoting.InvocationResponse = -5123635159962572581
      hash on field java.lang.String$sessionId = 7208879560699038148
      hash on field boolean$isException = 6944741943427072418
      hash on field java.lang.Object$result = -1910178885101376583
      hash on field java.util.Map$payload = -311608580347357596
      Reading object for id=1 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = org.jboss.remoting.InvocationResponse
      defaultRead::class org.jboss.remoting.InvocationResponse contains 1 slots
      defaultRead::slot[0]=org.jboss.remoting.InvocationResponse
      readSlotWithFields slot=org.jboss.remoting.InvocationResponse
      FieldName on Read=sessionId
      readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=sessionId<<-reading Object
      Reading string with utfSize=36 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=isException
      FieldName on Read=result
      readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=result<<-reading Object
      objectFromDescription::reading new definition
      Reading string with utfSize=43 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      hash on field org.jboss.aop.NotFoundInDispatcherException = -5393238603831683671
      lookupMethodOnHierarchy::class=org.jboss.aop.NotFoundInDispatcherException looking for readResolve
      lookupMethodOnHierarchy::currentClass=class org.jboss.aop.NotFoundInDispatcherException
      lookupMethodOnHierarchy::currentClass=class org.jboss.aop.DispatcherConnectException
      lookupMethodOnHierarchy::currentClass=class java.lang.Exception
      lookupMethodOnHierarchy::currentClass=class java.lang.Throwable
      lookupMethodOnHierarchy::class=org.jboss.aop.NotFoundInDispatcherException looking for writeReplace
      lookupMethodOnHierarchy::currentClass=class org.jboss.aop.NotFoundInDispatcherException
      lookupMethodOnHierarchy::currentClass=class org.jboss.aop.DispatcherConnectException
      lookupMethodOnHierarchy::currentClass=class java.lang.Exception
      lookupMethodOnHierarchy::currentClass=class java.lang.Throwable
      hash on field org.jboss.aop.NotFoundInDispatcherException = -5393238603831683671
      hash on field java.lang.Object$oid = 5891181997928938037
      hash on field org.jboss.aop.DispatcherConnectException = -2548359991833714121
      hash on field java.lang.Exception = 7600092213321646919
      hash on field java.lang.Throwable = -7779525601374451073
      hash on field java.lang.String$detailMessage = -4333528023742127755
      hash on field java.lang.Throwable$cause = 4080765070623846565
      hash on field [Ljava.lang.StackTraceElement;$stackTrace = -4842200784206624673
      Reading object for id=4 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = org.jboss.aop.NotFoundInDispatcherException
      defaultRead::class org.jboss.aop.NotFoundInDispatcherException contains 4 slots
      defaultRead::slot[0]=java.lang.Throwable
      readSlotWithFields slot=java.lang.Throwable
      FieldName on Read=detailMessage
      readSlotWithFields slot=java.lang.Throwable field=detailMessage<<-reading Object
      Reading string with utfSize=60 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=cause
      readSlotWithFields slot=java.lang.Throwable field=cause<<-reading Object
      objectFromDescription::reading circular definition reference=4
      FieldName on Read=stackTrace
      readSlotWithFields slot=java.lang.Throwable field=stackTrace<<-reading Object
      objectFromDescription::reading new definition
      Reading string with utfSize=30 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      hash on field [Ljava.lang.StackTraceElement; = -8287581384409355425
      lookupMethodOnHierarchy::class=[Ljava.lang.StackTraceElement; looking for readResolve
      lookupMethodOnHierarchy::currentClass=class [Ljava.lang.StackTraceElement;
      lookupMethodOnHierarchy::class=[Ljava.lang.StackTraceElement; looking for writeReplace
      lookupMethodOnHierarchy::currentClass=class [Ljava.lang.StackTraceElement;
      hash on field [Ljava.lang.StackTraceElement; = -8287581384409355425
      Reading object for id=7 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = [Ljava.lang.StackTraceElement;
      objectFromDescription::reading new definition
      Reading string with utfSize=27 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      hash on field java.lang.StackTraceElement = -5341604412408429699
      lookupMethodOnHierarchy::class=java.lang.StackTraceElement looking for readResolve
      lookupMethodOnHierarchy::currentClass=class java.lang.StackTraceElement
      lookupMethodOnHierarchy::class=java.lang.StackTraceElement looking for writeReplace
      lookupMethodOnHierarchy::currentClass=class java.lang.StackTraceElement
      hash on field java.lang.StackTraceElement = -5341604412408429699
      hash on field java.lang.String$declaringClass = -4020810388859008069
      hash on field java.lang.String$methodName = 2043831539205426629
      hash on field java.lang.String$fileName = -3886410644765885867
      hash on field int$lineNumber = 1246520928412557129
      Reading object for id=9 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement
      defaultRead::class java.lang.StackTraceElement contains 1 slots
      defaultRead::slot[0]=java.lang.StackTraceElement
      readSlotWithFields slot=java.lang.StackTraceElement
      FieldName on Read=declaringClass
      readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object
      Reading string with utfSize=24 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=methodName
      readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object
      Reading string with utfSize=6 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=fileName
      readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object
      Reading string with utfSize=15 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=lineNumber
      objectFromDescription::reading new definition
      Reading object for id=14 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement
      defaultRead::class java.lang.StackTraceElement contains 1 slots
      defaultRead::slot[0]=java.lang.StackTraceElement
      readSlotWithFields slot=java.lang.StackTraceElement
      FieldName on Read=declaringClass
      readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object
      Reading string with utfSize=56 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=methodName
      readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object
      FieldName on Read=fileName
      readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object
      Reading string with utfSize=31 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=lineNumber
      objectFromDescription::reading new definition
      Reading object for id=17 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement
      defaultRead::class java.lang.StackTraceElement contains 1 slots
      defaultRead::slot[0]=java.lang.StackTraceElement
      readSlotWithFields slot=java.lang.StackTraceElement
      FieldName on Read=declaringClass
      readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object
      Reading string with utfSize=32 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=methodName
      readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object
      FieldName on Read=fileName
      readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object
      Reading string with utfSize=18 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=lineNumber
      objectFromDescription::reading new definition
      Reading object for id=20 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement
      defaultRead::class java.lang.StackTraceElement contains 1 slots
      defaultRead::slot[0]=java.lang.StackTraceElement
      readSlotWithFields slot=java.lang.StackTraceElement
      FieldName on Read=declaringClass
      readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object
      Reading string with utfSize=48 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=methodName
      readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object
      Reading string with utfSize=17 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=fileName
      readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object
      Reading string with utfSize=17 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=lineNumber
      objectFromDescription::reading new definition
      Reading object for id=24 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement
      defaultRead::class java.lang.StackTraceElement contains 1 slots
      defaultRead::slot[0]=java.lang.StackTraceElement
      readSlotWithFields slot=java.lang.StackTraceElement
      FieldName on Read=declaringClass
      readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object
      FieldName on Read=methodName
      readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object
      Reading string with utfSize=5 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=fileName
      readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object
      FieldName on Read=lineNumber
      objectFromDescription::reading new definition
      Reading object for id=26 classLoader=[ThreadContextLoader, current context: [ClassLoader: [[C:\OraHome_1\j2ee\OC4J_BPEL\applications\orabpel\ejb_ob_engine.jar archive], [C:\OraHome_1\j2ee\OC4J_BPEL\application-deployments\orabpel\ejb_ob_engine.jar/deployment.cache]]]] className = java.lang.StackTraceElement
      defaultRead::class java.lang.StackTraceElement contains 1 slots
      defaultRead::slot[0]=java.lang.StackTraceElement
      readSlotWithFields slot=java.lang.StackTraceElement
      FieldName on Read=declaringClass
      readSlotWithFields slot=java.lang.StackTraceElement field=declaringClass<<-reading Object
      FieldName on Read=methodName
      readSlotWithFields slot=java.lang.StackTraceElement field=methodName<<-reading Object
      Reading string with utfSize=3 isLong=false
      readString::pulling data to Buffer at pos 0 size= 0
      FieldName on Read=fileName
      readSlotWithFields slot=java.lang.StackTraceElement field=fileName<<-reading Object
      FieldName on Read=lineNumber
      defaultRead::slot[1]=java.lang.Exception
      readSlotWithFields slot=java.lang.Exception
      defaultRead::slot[2]=org.jboss.aop.DispatcherConnectException
      readSlotWithFields slot=org.jboss.aop.DispatcherConnectException
      defaultRead::slot[3]=org.jboss.aop.NotFoundInDispatcherException
      readSlotWithFields slot=org.jboss.aop.NotFoundInDispatcherException
      FieldName on Read=oid
      readSlotWithFields slot=org.jboss.aop.NotFoundInDispatcherException field=oid<<-reading Object
      FieldName on Read=payload
      readSlotWithFields slot=org.jboss.remoting.InvocationResponse field=payload<<-reading Object
      Caught Exception:
      org.jboss.aop.NotFoundInDispatcherException: Object with oid: -2147483579 was not found in the Dispatcher
       at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:85)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:127)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:715)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:552)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:377)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:159)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
       at org.jboss.remoting.Client.invoke(Client.java:1414)
       at org.jboss.remoting.Client.invoke(Client.java:511)
       at org.jboss.jms.client.delegate.DelegateSupport.invoke(DelegateSupport.java:111)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:182)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:69)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate$stop_N2856118408655404442.invokeNext(ClientConnectionDelegate$stop_N2856118408655404442.java)
       at org.jboss.jms.client.delegate.ClientConnectionDelegate.stop(ClientConnectionDelegate.java)
       at org.jboss.jms.client.JBossConnection.stop(JBossConnection.java:126)
       at oracle.tip.adapter.jms.JMS.JMSConnection.stop(JMSConnection.java:432)
       at oracle.tip.adapter.jms.JMS.JMSMessageConsumer.stop(JMSMessageConsumer.java:252)
       at oracle.tip.adapter.jms.inbound.JmsConsumer.stop(JmsConsumer.java:164)
       at oracle.tip.adapter.jms.JmsEndpoint.stop(JmsEndpoint.java:150)
       at oracle.tip.adapter.jms.JmsResourceAdapter.endpointDeactivation(JmsResourceAdapter.java:196)
       at oracle.tip.adapter.fw.jca.AdapterFrameworkImpl.endpointDeactivation(AdapterFrameworkImpl.java:590)
       at oracle.tip.adapter.fw.agent.jca.JCAActivationAgent.deactivateInboundJcaEndpoint(JCAActivationAgent.java:953)
       at oracle.tip.adapter.fw.agent.jca.JCAActivationAgent.uninit(JCAActivationAgent.java:458)
       at com.collaxa.cube.engine.core.BaseCubeProcess.disableActivationAgents(BaseCubeProcess.java:516)
       at com.collaxa.cube.engine.core.BaseCubeProcess.uninit(BaseCubeProcess.java:321)
       at com.collaxa.cube.engine.deployment.CubeProcessLoader.destroy(CubeProcessLoader.java:589)
       at com.collaxa.cube.engine.deployment.CubeProcessHolder.uninit(CubeProcessHolder.java:141)
       at com.collaxa.cube.engine.CubeEngine.uninit(CubeEngine.java:794)
       at com.collaxa.cube.admin.ServerManager.uninit(ServerManager.java:334)
       at com.collaxa.cube.admin.ServerManager$1.run(ServerManager.java:150)
      


      Any ideas on where to go from here to get some possible idea what may be causing this error?