2 Replies Latest reply on Dec 27, 2010 2:50 PM by Igor HVR

    Help with unexpected HornetQ crash.

    Joe Schmoe Newbie

      After having a couple of appservers using HornetQ up and running since July 2, one of them just crashed, and I'm not sure how to go about figuring out why.  Does anyone have any advice?

       

      Here is the very first error I got in the logs:

       

      Jul 8, 2010 10:22:27 AM org.hornetq.core.logging.impl.JULLogDelegate warn
      WARNING: Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@1155ec36[local= /127.0.0.1:51765, remote
      =localhost/127.0.0.1:5445] [code=3]
      Exception in thread "Thread-3" java.lang.NullPointerException
              at com.lf.service.jms.JMSQueueManagementInfoBackgroundLogger.logMessageCounterInfo(Unknown Source)
              at com.lf.service.jms.JMSQueueManagementInfoBackgroundLogger$BackgroundLogger.run(Unknown Source)
              at java.lang.Thread.run(Thread.java:619)
      Exception in thread "Thread-4" java.lang.NullPointerException
              at com.lf.service.jms.JMSTopicManagementInfo.getMessageCount(Unknown Source)
              at com.lf.service.jms.JMSTopicManagementInfoBackgroundLogger.logMessageCounterInfo(Unknown Source)
              at com.lf.service.jms.JMSTopicManagementInfoBackgroundLogger$BackgroundLogger.run(Unknown Source)
              at java.lang.Thread.run(Thread.java:619)
      Jul 8, 2010 10:24:01 AM org.hornetq.core.logging.impl.JULLogDelegate error
      SEVERE: Failed to handle failover
      HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 32]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
              at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:874)
              at org.hornetq.core.client.impl.FailoverManagerImpl.reconnectSessions(FailoverManagerImpl.java:818)
              at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:719)
              at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:581)
              at org.hornetq.core.client.impl.FailoverManagerImpl.access$600(FailoverManagerImpl.java:73)
              at org.hornetq.core.client.impl.FailoverManagerImpl$DelegatingFailureListener.connectionFailed(FailoverManagerImpl.java:1151)
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:482)
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)
              at org.hornetq.core.client.impl.FailoverManagerImpl$PingRunnable$1.run(FailoverManagerImpl.java:1209)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:619)
      Jul 8, 2010 10:24:01 AM org.hornetq.core.logging.impl.JULLogDelegate warn
      WARNING: Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@6a2f4aa7[local= /127.0.0.1:43121, remote
      =localhost/127.0.0.1:5445] [code=3]

       

      And later in the chain:

      Jul 8, 2010 10:27:31 AM org.hornetq.core.logging.impl.JULLogDelegate error

      SEVERE: Failed to handle failover

      java.lang.ClassCastException: org.hornetq.core.protocol.core.impl.wireformat.CreateSessionResponseMessage cannot be cast to org.hornetq.core.protocol.core.impl.wireformat.Reatta

      chSessionResponseMessage

              at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:874)

              at org.hornetq.core.client.impl.FailoverManagerImpl.reconnectSessions(FailoverManagerImpl.java:818)

              at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:719)

              at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:581)

              at org.hornetq.core.client.impl.FailoverManagerImpl.access$600(FailoverManagerImpl.java:73)

              at org.hornetq.core.client.impl.FailoverManagerImpl$DelegatingFailureListener.connectionFailed(FailoverManagerImpl.java:1151)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:482)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)

              at org.hornetq.core.client.impl.FailoverManagerImpl$PingRunnable$1.run(FailoverManagerImpl.java:1209)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

      Jul 8, 2010 10:27:31 AM org.hornetq.core.logging.impl.JULLogDelegate error

      SEVERE: Caught exception

      HornetQException[errorCode=101 message=Queue f040792d-d3a2-44a2-8098-2b0e4fcb6a91 already exists]

              at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:1307)

              at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:679)

              at org.hornetq.core.server.impl.ServerSessionImpl.createQueue(ServerSessionImpl.java:354)

              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:251)

              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:471)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:451)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:412)

              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:67)

              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)

              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)

              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)

              at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:340)

              at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:272)

              at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)

              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

              at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

       

      Here is my hornetq configuration:

       

      <configuration xmlns="urn:hornetq"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                     xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
            
           <journal-min-files>10</journal-min-files>
           
           <jmx-management-enabled>true</jmx-management-enabled>
           <management-address>jms.queue.hornetq.management</management-address>
           <message-counter-enabled>true</message-counter-enabled> 
           <message-counter-sample-period>30000</message-counter-sample-period> 
           <message-counter-max-day-history>30</message-counter-max-day-history> 
         
         <connectors>
            <connector name="netty">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
               <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
               <param key="use-nio" value="true" />
               <param key="tcp-send-buffer-size" value="1048576" />
               <param key="tcp-receive-buffer-size" value="1048576" />
            </connector>
            
            <connector name="netty-throughput">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
               <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
               <param key="batch-delay" value="50"/>
               <param key="use-nio" value="true" />
               <param key="tcp-send-buffer-size" value="1048576" />
               <param key="tcp-receive-buffer-size" value="1048576" />
            </connector>
         </connectors>
      
         <acceptors>
            <acceptor name="netty">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
               <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
               <param key="use-nio" value="true" />
               <param key="tcp-send-buffer-size" value="1048576" />
               <param key="tcp-receive-buffer-size" value="1048576" />
            </acceptor>
            
            <acceptor name="netty-throughput">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
               <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
               <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
               <param key="batch-delay" value="50"/>
               <param key="direct-deliver" value="false"/>
               <param key="use-nio" value="true" />
               <param key="tcp-send-buffer-size" value="1048576" />
               <param key="tcp-receive-buffer-size" value="1048576" />
            </acceptor>
         </acceptors>
      
         <security-settings>
                <security-setting match="jms.queue.hornetq.management">
                          <permission type="manage" roles="guest" />
                </security-setting>
                   
            <security-setting match="#">
               <permission type="createDurableQueue" roles="guest" />
               <permission type="deleteDurableQueue" roles="guest" />
               <permission type="createNonDurableQueue" roles="guest" />
               <permission type="deleteNonDurableQueue" roles="guest" />
               <permission type="createDurableTopic" roles="guest" />
               <permission type="deleteDurableTopic" roles="guest" />
               <permission type="createNonDurableTopic" roles="guest" />
               <permission type="deleteNonDurableTopic" roles="guest" />
               <permission type="consume" roles="guest" />
               <permission type="send" roles="guest" />
            </security-setting>
         </security-settings>
      
         <address-settings>
         
                   <address-setting match="jms.queue.toolbarEventQueue">
                     <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                     <max-size-bytes>104857600</max-size-bytes>
                     <page-size-bytes>10485760</page-size-bytes> 
                     <address-full-policy>PAGE</address-full-policy>
                </address-setting>
         
                <address-setting match="jms.topic.toolbarMessageRetrievalTopic">
                     <last-value-queue>true</last-value-queue>
                     <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                     <max-size-bytes>26214400</max-size-bytes> 
                     <address-full-policy>DROP</address-full-policy>
                </address-setting>
                
                <address-setting match="jms.queue.toolbarCommandQueue">
                     <last-value-queue>true</last-value-queue>
                     <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                    <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                     <max-size-bytes>26214400</max-size-bytes> 
                     <address-full-policy>DROP</address-full-policy>
                </address-setting>
                
                <address-setting match="jms.queue.DLQ">
                     <max-size-bytes>10485760</max-size-bytes>
                     <address-full-policy>DROP</address-full-policy>
                </address-setting>
                
                <address-setting match="jms.queue.ExpiryQueue">
                     <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                     <max-size-bytes>10485760</max-size-bytes> 
                     <address-full-policy>DROP</address-full-policy>
                </address-setting>
                   
            <!--default for catch all-->
            <address-setting match="#">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
               <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <redelivery-delay>0</redelivery-delay>
               <max-size-bytes>26214400</max-size-bytes>
               <page-size-bytes>10485760</page-size-bytes>       
               <message-counter-history-day-limit>30</message-counter-history-day-limit>
               <address-full-policy>PAGE</address-full-policy>
            </address-setting>
         </address-settings>
      
      </configuration>
      

       

      And my hornetq jms configuration:

       

      <configuration xmlns="urn:hornetq"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">
           
           <connection-factory name="NettyConnectionFactory">
              <connectors>
               <connector-ref connector-name="netty"/>
              </connectors>
                <entries>
                     <entry name="java:/ConnectionFactory"/>
                     <entry name="/XAConnectionFactory"/>
                </entries>
                <retry-interval>2000</retry-interval>
                <retry-interval-multiplier>1</retry-interval-multiplier>
                <max-retry-interval>2000</max-retry-interval>
                <reconnect-attempts>1000</reconnect-attempts>
           </connection-factory>
           
           <connection-factory name="NettyThroughputConnectionFactory">
              <connectors>
               <connector-ref connector-name="netty-throughput"/>
              </connectors>
                <entries>
                     <entry name="java:/ThroughputConnectionFactory"/>
                     <entry name="/XAThroughputConnectionFactory"/>
                </entries>
                <retry-interval>2000</retry-interval>
                <retry-interval-multiplier>1</retry-interval-multiplier>
                <max-retry-interval>2000</max-retry-interval>
                <reconnect-attempts>1000</reconnect-attempts>
           </connection-factory>
           
           <!-- DEFAULT QUEUES -->
           <queue name="DLQ">
                <entry name="/queue/DLQ"/>
           </queue>
           <queue name="ExpiryQueue">
                <entry name="/queue/ExpiryQueue"/>
           </queue>   
           
           <!-- Toolbar Event Logging Queue -->
              <queue name="toolbarEventQueue">
                <entry name="/queue/toolbarEventQueue" />
           </queue>
           
           <!-- Toolbar Command Queue -->
              <queue name="toolbarCommandQueue">
                <entry name="/queue/toolbarCommandQueue" />
           </queue>
           
           <!-- Toolbar Message Retrieval Topic -->
              <topic name="toolbarMessageRetrievalTopic">
                <entry name="/topic/toolbarMessageRetrievalTopic" />
           </topic>
      
      </configuration>
      
       
      
        • 1. Re: Help with unexpected HornetQ crash.
          Joe Schmoe Newbie

          Some other exceptions:

          javax.jms.JMSException: Timed out waiting for response when sending packet 71
                 at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
                 at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:285)
                 at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
                 at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
                 at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)

          ...

           

          and...

          javax.jms.JMSException: Timed out waiting for response when sending packet 43

                 at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
                 at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:514)
                 at org.hornetq.core.client.impl.DelegatingSession.commit(DelegatingSession.java:156)
                 at org.hornetq.jms.client.HornetQMessage.acknowledge(HornetQMessage.java:857)

           

           

          It seems that the HornetQ server itself crashed since all the producers and consumers seem to be timing out waiting for a response.

          • 2. Re: Help with unexpected HornetQ crash.
            Igor HVR Newbie

            Your best bet is to increase the callTimeout in the ClientSessionFactory you use. If the server is just temporarily overloaded this will give your client sessions more time before giving up and throwing an exception (which will make things even worse in the server side).

             

            I had this and similar exceptions fairly frequently (a couple of times every day - however when it happened the error bubbled up and crashed the application) in my production setup - after increasing my timeout to five minutes (the default is 30 seconds), I never saw any errors like this again.