4 Replies Latest reply on Jan 17, 2011 10:26 AM by braym

    MessageSucker failure

    chipschoch

      JBossAS 4.2.2, JBM 1.4.4

      Running in out production system with two servers we are regularly getting this error. I have no idea what is causing it, but we never saw it until we upgraded to 1.4.4. This is killing us. Any ideas?

      2009-09-16 14:29:48,726 ERROR | ReusableThread | org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager | Failed to process notifica
      tion
      org.jboss.jms.exception.MessagingJMSException: Failed to invoke
       at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$createConsumerDelegate$aop(ClientSessionDel
      egate.java:267)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDeleg
      ate_8721389917985689973.java)
       at org.jboss.jms.client.container.StateCreationAspect.handleCreateConsumerDelegate(StateCreationAspect.java:136)
       at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect30.invoke(StateCreationAspect30.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDeleg
      ate_8721389917985689973.java)
       at org.jboss.jms.client.container.ConsumerAspect.handleCreateConsumerDelegate(ConsumerAspect.java:76)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ConsumerAspect29.invoke(ConsumerAspect29.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDeleg
      ate_8721389917985689973.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDeleg
      ate_8721389917985689973.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$createConsumerDelegate_8721389917985689973.invokeNext(ClientSessionDelegate$createConsumerDeleg
      ate_8721389917985689973.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.createConsumerDelegate(ClientSessionDelegate.java)
       at org.jboss.messaging.core.impl.clusterconnection.MessageSucker.start(MessageSucker.java:126)
       at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.createSucker(ClusterConnectionManager.java:478)
       at org.jboss.messaging.core.impl.clusterconnection.ClusterConnectionManager.notify(ClusterConnectionManager.java:345)
       at org.jboss.messaging.core.impl.DefaultClusterNotifier.sendNotification(DefaultClusterNotifier.java:72)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBindingInMemory(MessagingPostOffice.java:2447)
       at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBindingFromCluster(MessagingPostOffice.java:1036)
       at org.jboss.messaging.core.impl.postoffice.BindRequest.execute(BindRequest.java:55)
       at org.jboss.messaging.core.impl.postoffice.GroupMember$ControlRequestHandler.handle(GroupMember.java:628)
       at org.jgroups.blocks.MessageDispatcher.handle(MessageDispatcher.java:610)
       at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:654)
       at org.jgroups.blocks.RequestCorrelator.access$200(RequestCorrelator.java:40)
       at org.jgroups.blocks.RequestCorrelator$Request.run(RequestCorrelator.java:944)
       at org.jgroups.util.ReusableThread.run(ReusableThread.java:234)
       at java.lang.Thread.run(Thread.java:595)
      Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id r-xoi7rmzf-1-tmr6rmzf-kuxebg-mwxa
       at org.jboss.jms.wireformat.SessionCreateConsumerDelegateRequest.serverInvoke(SessionCreateConsumerDelegateRequest.java:97)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:862)
       at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:609)
       at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:421)
       at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:174)
       at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:182)
       at org.jboss.remoting.Client.invoke(Client.java:1858)
       at org.jboss.remoting.Client.invoke(Client.java:718)
       at org.jboss.remoting.Client.invoke(Client.java:706)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)


        • 1. Re: MessageSucker failure
          gaohoward

          This probably because the server side has closed related JMS object (as the id indicated) due to remoting lease time out.

          Please try to enlarge the clientLeasePeriod in the remoting configure file.

          • 2. Re: MessageSucker failure
            chipschoch

            Thanks for the tip. Any clues on how to determine what a reasonable value for this is?

            • 3. Re: MessageSucker failure
              gaohoward

              About the value, I think it depends on your network's latency.
              Ask remoting team, they have the expertise.

              • 4. Re: MessageSucker failure
                braym

                We have Jboss 5.1.0GA with JBM 1.4.3GA in cluster configuration (4 nodes). Rarely we take the exception attached, and after that, messanging system on the node, die.

                In addition to enlarge clientLeasePeriod value, it make sense to enlarge validatorPingPeriod and validatorPingTimeout values?

                Is it possible that a fullGC can generate this raise condition?

                Is it possible a mutual interaction with Jgroups setting? (we have clustered queue)

                Installing JBM 1.4.7GA and new remoting package, is it possible resolve this situation?

                When you talk about network's latency, did you refer a specific parameters?

                 

                Thank you!