13 Replies Latest reply on Dec 25, 2012 7:12 AM by ohmygod

    HornetQ error

    ohmygod

      Anybody has an idea how to fix this kind of error? I am not sure how it happens but it happens.

       

       

      Here is the log trace.

       

       

      012-12-21 19:32:52,436;[Thread-2041 (HornetQ-client-global-threads-1771493528)];ERROR;org.hornetq.ra.inflow.HornetQMessageHandler;Failed to deliver message: HornetQException[errorCode=102 message=Session is closed]
       at org.hornetq.core.client.impl.ClientSessionImpl.checkClosed(ClientSessionImpl.java:1881)
       at org.hornetq.core.client.impl.ClientSessionImpl.acknowledge(ClientSessionImpl.java:758)
       at org.hornetq.core.client.impl.ClientConsumerImpl.doAck(ClientConsumerImpl.java:1101)
       at org.hornetq.core.client.impl.ClientConsumerImpl.acknowledge(ClientConsumerImpl.java:729)
       at org.hornetq.core.client.impl.ClientMessageImpl.acknowledge(ClientMessageImpl.java:98)
       at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:282)
       at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:983)
       at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48)
       at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1113)
       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:662)
      
      

       

      I am using JBoss 7.1.1.

        • 1. Re: HornetQ error
          ohmygod

          Also met this kind of warning message today.

           

          2012-12-21 20:43:05,652;[Periodic Recovery];WARN ;org.hornetq.jms.server.recovery.HornetQXAResourceWrapper;null: javax.transaction.xa.XAException
                          at org.hornetq.core.client.impl.ClientSessionImpl.recover(ClientSessionImpl.java:1528)
                          at org.hornetq.core.client.impl.DelegatingSession.recover(DelegatingSession.java:448)
                          at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.recover(HornetQXAResourceWrapper.java:81)
                          at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:503)
                          at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471)
                          at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385)
                          at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166)
                          at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789)
                          at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371)
          
          
          
          • 2. Re: HornetQ error
            ohmygod

            I am not using any acknowledge() method in my MDB class. Why it is calling it in the log trace?

            • 3. Re: HornetQ error
              ohmygod

              How about this one?

               

               

              2012-12-24 14:16:12,931;[Thread-178 (HornetQ-client-global-threads-1302089035)];ERROR;org.hornetq.core.client.impl.ClientSessionImpl;Caught Exception : HornetQException[errorCode=2 message=Connection is destroyed]
                        at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:233)
                        at org.hornetq.core.client.impl.ClientSessionImpl.end(ClientSessionImpl.java:1365)
                        at org.hornetq.core.client.impl.DelegatingSession.end(DelegatingSession.java:333)
                        at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:597)
                        at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2283)
                        at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1466)
                        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)
                        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
                        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
                        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
                        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
                        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery(MessageEndpointInvocationHandler.java:72)
                        at sun.reflect.GeneratedMethodAccessor411.invoke(Unknown Source)
                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                        at java.lang.reflect.Method.invoke(Method.java:597)
                        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(AbstractInvocationHandler.java:60)
                        at org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(MessageEndpointInvocationHandler.java:136)
                        at org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:73)
                        at $Proxy13.afterDelivery(Unknown Source)
                        at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:287)
                        at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:983)
                        at org.hornetq.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:48)
                        at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1113)
                        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                        at java.lang.Thread.run(Thread.java:619)
              
              
              
              • 4. Re: HornetQ error
                gaohoward

                Can you describe a little how your test works? or better yet do you have a test case that can be uploaded here?

                • 5. Re: HornetQ error
                  ohmygod

                  I am using JBoss as 7.1.1 and there are a lot of MDB classes with mostly like below configuration in the head of the class to handle requests.

                   

                   

                  @MessageDriven(messageListenerInterface = MessageListener.class, activationConfig =
                  {
                          @ActivationConfigProperty(propertyName = "destination", propertyValue = "ForExtractor"),
                          @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Topic"),
                          @ActivationConfigProperty(propertyName = "clientID", propertyValue = "ExtractorMDB"),
                          @ActivationConfigProperty(propertyName = "subscriptionDurability", propertyValue = "Durable") })
                  @TransactionManagement(value = TransactionManagementType.CONTAINER)
                  

                   

                  These kind of errors and warnings happen when there are a lot of requests being handled at the same time (should be more than 100 requests).

                   

                  Below is the message creator code (basically normal codes)

                   

                  try
                  {
                              Context context = s_appServer.getNamingContext();
                              TopicConnectionFactory cf = (TopicConnectionFactory) context
                                      .lookup(JMS_TOPIC_FACTORY_NAME);
                              TopicConnection topicConnection = cf.createTopicConnection();
                              topicConnection.start();
                              TopicSession session = topicConnection.createTopicSession(false,
                                      Session.AUTO_ACKNOWLEDGE); 
                                                Topic topic = (Topic) context.lookup(jmsTopicName);
                                          TopicPublisher sender = session.createPublisher(topic);
                                  ObjectMessage om = session.createObjectMessage(p_cxeMessage);
                    
                  }
                  finally
                  {
                                    if (session != null)
                              {
                                  try
                                  {
                                      session.close();
                                  }
                                  catch (Exception ex)
                                  {
                                  }
                              }
                  }
                  
                  
                  • 6. Re: HornetQ error
                    ohmygod

                    For small number of requests, there is no any error and warning, just work fine. So I suspect this is some performance issue of Hornetq. Are there any settings that can be applied into standalone.xml in JBoss 7.1.1?

                    • 7. Re: HornetQ error
                      gaohoward

                      Hi,

                       

                      Looks like some connection are destroyed too early. We have fixed a bunch of issues since 7.1.1. Would you mind trying the latest version ?

                       

                      Thanks

                      Howard

                      • 8. Re: HornetQ error
                        ohmygod

                        Thanks for the reply. Is there patch than can be applied to 7.1.1 version? Changing the entire project to a new version may take a lot of effort thant we cannot afford.

                        • 9. Re: HornetQ error
                          xin.zhang.song

                          Hi Yong Hao

                           

                          Can you post me the seemingly issues and fixed version?

                           

                          Thanks

                           

                          Xin

                          • 10. Re: HornetQ error
                            gaohoward

                            Hi Mike and Xin,

                             

                            I'm not sure what's the issue you are having based on the information given here. If you can provide some tests to reproduce it I'd like to do further investigation.

                             

                            Howard

                            • 11. Re: HornetQ error
                              ohmygod

                              Hi Yong Hao,

                               

                              I wonder if there is any hint for you from the error/warn message because it is pretty hard to reproduce this problem (need a lot of requests to MDB at the same time). Is there anything we can do to configure to avoid this kind of cases? such as below warn message from one of my ther other thread https://community.jboss.org/message/784823#784823

                               

                               

                              2012-12-21 19:33:02,789;[hornetq-failure-check-thread];WARN ;org.hornetq.core.protocol.core.impl.RemotingConnectionImpl;Connection failure has been detected: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
                              

                               

                              The message above talks about the option "connection-ttl" and "client-failure-check-period". I am concerned if these two cases have some connections. Please advise.

                              • 12. Re: HornetQ error
                                gaohoward

                                Did you try to increase the 'connecton-ttl' and see if that make any difference?

                                • 13. Re: HornetQ error
                                  ohmygod

                                  I put the option under the "

                                  <pooled-connection-factory name="hornetq-ra">" element and "

                                  <connection-factory name="RemoteConnectionFactory">" element

                                  in below setting, but seems not working (still see the warn/error message after some time with a lot of requests coming into MDB).

                                   

                                  <jms-connection-factories>
                                                      <connection-factory name="InVmConnectionFactory">
                                                          <connectors>
                                                              <connector-ref connector-name="in-vm"/>
                                                          </connectors>
                                                          <entries>
                                                              <entry name="java:/ConnectionFactory"/>
                                                          </entries>
                                                      </connection-factory>
                                                      <connection-factory name="RemoteConnectionFactory">
                                                          <connectors>
                                                              <connector-ref connector-name="netty"/>
                                                          </connectors>
                                                          <entries>
                                                              <entry name="RemoteConnectionFactory"/>
                                                              <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                                                              <entry name="JMSConnectionFactory"/>
                                                              <entry name="java:jboss/exported/jms/JMSConnectionFactory"/>
                                                          </entries>
                                                      </connection-factory>
                                                      <pooled-connection-factory name="hornetq-ra">
                                                          <transaction mode="xa"/>
                                                          <connectors>
                                                              <connector-ref connector-name="in-vm"/>
                                                          </connectors>
                                                          <entries>
                                                              <entry name="java:/JmsXA"/>
                                                          </entries>
                                                      </pooled-connection-factory>
                                                  </jms-connection-factories>
                                  

                                   

                                  This is done in another thread https://community.jboss.org/message/784877#784877

                                   

                                  Can you suggest where to put the two options in standalone.xml of jboss as 7?