JBM exception
theoverlord Aug 1, 2008 2:11 PMI'm using JBM 1.4.0.SP3_CP03 to send messages to a remote MDB.
I'm using a clustered queue.
After a sending many messages, JBM basically dies and starts spewing the following messages repeatedly:
2008-08-01 11:00:02,851 [ReusableThread] ERROR org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager - Failed to process notification org.jboss.jms.exception.MessagingJMSException: Failed to invoke at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDelegate.java:245) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:142) at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect30.invoke(StateCreationAspect30.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:73) at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect29.invoke(ConsumerAspect29.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDelegate_8721389917985689973.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java) at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.start(MessageSucker.java:126) at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.createSucker(ClusterConnectionManager.java:478) at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.notify(ClusterConnectionManager.java:318) at org.jboss.messaging.core.impl.DefaultClusterNotifier.sendNotification(DefaultClusterNotifier.java:72) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBindingInMemory(MessagingPostOffice.java:2427) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalAddBinding(MessagingPostOffice.java:1868) at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBindingFromCluster(MessagingPostOffice.java:1035) at org.jboss.messaging.core.impl.postoffice.BindRequest.execute(BindRequest.java:55) at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:606) at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:610) at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654) at org.jgroups.blocks.RequestCorrelator.access$200(RequestCorrelator.java:40) at org.jgroups.blocks.RequestCorrelator$Request.run(RequestCorrelator.java:944) at org.jgroups.util.ReusableThread.run(ReusableThread.java:234) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id i-nueykcjf-1-fkmxkcjf-7pmlak-t36hs4 at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:97) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608) at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420) at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173) at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163) at org.jboss.remoting.Client.invoke(Client.java:1634) at org.jboss.remoting.Client.invoke(Client.java:548) at org.jboss.remoting.Client.invoke(Client.java:536) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189) ... 31 more
After it enters this state, it's useless and unrecoverable. It requires a total cluster restart of the AS.
On the MDB side (different physical machine), it's spewing this:
INFO org.jboss.jms.client.FailoverCommandCenter - JBoss Messaging server failure detected - waiting for failover to complete... 2008-08-01 08:46:03,919 [Thread-273] WARN org.jboss.remoting.LeasePinger - LeasePinger[SocketClientInvoker[9fa2fb, bisocket://apr-s02:8527](4sh63s-mvdw0q-fjckz12o-1-fjcl06j5-5)] tried to remove lease for client with session ID 4sh63s-mvdw0q-fjckz12o-1-fjcyu0zf-5ob7, but no such lease was found
The key to this mystery is that it doesn't happen when I move the MDB into the same machines where the queue reside.
When I move the MDB external to the queues, that's when the problem occurs.
I've tried hundreds of different things to figure this out but it all fails after several hours of stress testing.
Any ideas what is going on?