2 Replies Latest reply on Dec 27, 2010 2:50 PM by igorhvr

Help with unexpected HornetQ crash.

unsavory Jul 8, 2010 2:22 PM

After having a couple of appservers using HornetQ up and running since July 2, one of them just crashed, and I'm not sure how to go about figuring out why. Does anyone have any advice?

Here is the very first error I got in the logs:

Jul 8, 2010 10:22:27 AM org.hornetq.core.logging.impl.JULLogDelegate warn

WARNING: Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@1155ec36[local= /127.0.0.1:51765, remote

=localhost/127.0.0.1:5445] [code=3]

Exception in thread "Thread-3" java.lang.NullPointerException

        at com.lf.service.jms.JMSQueueManagementInfoBackgroundLogger.logMessageCounterInfo(Unknown Source)

        at com.lf.service.jms.JMSQueueManagementInfoBackgroundLogger$BackgroundLogger.run(Unknown Source)

        at java.lang.Thread.run(Thread.java:619)

Exception in thread "Thread-4" java.lang.NullPointerException

        at com.lf.service.jms.JMSTopicManagementInfo.getMessageCount(Unknown Source)

        at com.lf.service.jms.JMSTopicManagementInfoBackgroundLogger.logMessageCounterInfo(Unknown Source)

        at com.lf.service.jms.JMSTopicManagementInfoBackgroundLogger$BackgroundLogger.run(Unknown Source)

        at java.lang.Thread.run(Thread.java:619)

Jul 8, 2010 10:24:01 AM org.hornetq.core.logging.impl.JULLogDelegate error

SEVERE: Failed to handle failover

HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 32]

        at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)

        at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:874)

        at org.hornetq.core.client.impl.FailoverManagerImpl.reconnectSessions(FailoverManagerImpl.java:818)

        at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:719)

        at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:581)

        at org.hornetq.core.client.impl.FailoverManagerImpl.access$600(FailoverManagerImpl.java:73)

        at org.hornetq.core.client.impl.FailoverManagerImpl$DelegatingFailureListener.connectionFailed(FailoverManagerImpl.java:1151)

        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:482)

        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)

        at org.hornetq.core.client.impl.FailoverManagerImpl$PingRunnable$1.run(FailoverManagerImpl.java:1209)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:619)

Jul 8, 2010 10:24:01 AM org.hornetq.core.logging.impl.JULLogDelegate warn

WARNING: Connection failure has been detected: Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@6a2f4aa7[local= /127.0.0.1:43121, remote

=localhost/127.0.0.1:5445] [code=3]

And later in the chain:

Jul 8, 2010 10:27:31 AM org.hornetq.core.logging.impl.JULLogDelegate error
SEVERE: Failed to handle failover
java.lang.ClassCastException: org.hornetq.core.protocol.core.impl.wireformat.CreateSessionResponseMessage cannot be cast to org.hornetq.core.protocol.core.impl.wireformat.Reatta
chSessionResponseMessage
        at org.hornetq.core.client.impl.ClientSessionImpl.handleFailover(ClientSessionImpl.java:874)
        at org.hornetq.core.client.impl.FailoverManagerImpl.reconnectSessions(FailoverManagerImpl.java:818)
        at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:719)
        at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:581)
        at org.hornetq.core.client.impl.FailoverManagerImpl.access$600(FailoverManagerImpl.java:73)
        at org.hornetq.core.client.impl.FailoverManagerImpl$DelegatingFailureListener.connectionFailed(FailoverManagerImpl.java:1151)
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:482)
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:254)
        at org.hornetq.core.client.impl.FailoverManagerImpl$PingRunnable$1.run(FailoverManagerImpl.java:1209)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
Jul 8, 2010 10:27:31 AM org.hornetq.core.logging.impl.JULLogDelegate error
SEVERE: Caught exception
HornetQException[errorCode=101 message=Queue f040792d-d3a2-44a2-8098-2b0e4fcb6a91 already exists]
        at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:1307)
        at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:679)
        at org.hornetq.core.server.impl.ServerSessionImpl.createQueue(ServerSessionImpl.java:354)
        at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:251)
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:471)
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:451)
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:412)
        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:67)
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:340)
        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:272)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:192)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

Here is my hornetq configuration:

<configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
      
     <journal-min-files>10</journal-min-files>
     
     <jmx-management-enabled>true</jmx-management-enabled>
     <management-address>jms.queue.hornetq.management</management-address>
     <message-counter-enabled>true</message-counter-enabled> 
     <message-counter-sample-period>30000</message-counter-sample-period> 
     <message-counter-max-day-history>30</message-counter-max-day-history> 
   
   <connectors>
      <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
         <param key="use-nio" value="true" />
         <param key="tcp-send-buffer-size" value="1048576" />
         <param key="tcp-receive-buffer-size" value="1048576" />
      </connector>
      
      <connector name="netty-throughput">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
         <param key="batch-delay" value="50"/>
         <param key="use-nio" value="true" />
         <param key="tcp-send-buffer-size" value="1048576" />
         <param key="tcp-receive-buffer-size" value="1048576" />
      </connector>
   </connectors>

   <acceptors>
      <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
         <param key="use-nio" value="true" />
         <param key="tcp-send-buffer-size" value="1048576" />
         <param key="tcp-receive-buffer-size" value="1048576" />
      </acceptor>
      
      <acceptor name="netty-throughput">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port"  value="${hornetq.remoting.netty.batch.port:5455}"/>
         <param key="batch-delay" value="50"/>
         <param key="direct-deliver" value="false"/>
         <param key="use-nio" value="true" />
         <param key="tcp-send-buffer-size" value="1048576" />
         <param key="tcp-receive-buffer-size" value="1048576" />
      </acceptor>
   </acceptors>

   <security-settings>
          <security-setting match="jms.queue.hornetq.management">
                    <permission type="manage" roles="guest" />
          </security-setting>
             
      <security-setting match="#">
         <permission type="createDurableQueue" roles="guest" />
         <permission type="deleteDurableQueue" roles="guest" />
         <permission type="createNonDurableQueue" roles="guest" />
         <permission type="deleteNonDurableQueue" roles="guest" />
         <permission type="createDurableTopic" roles="guest" />
         <permission type="deleteDurableTopic" roles="guest" />
         <permission type="createNonDurableTopic" roles="guest" />
         <permission type="deleteNonDurableTopic" roles="guest" />
         <permission type="consume" roles="guest" />
         <permission type="send" roles="guest" />
      </security-setting>
   </security-settings>

   <address-settings>
   
             <address-setting match="jms.queue.toolbarEventQueue">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
              <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <max-size-bytes>104857600</max-size-bytes>
               <page-size-bytes>10485760</page-size-bytes> 
               <address-full-policy>PAGE</address-full-policy>
          </address-setting>
   
          <address-setting match="jms.topic.toolbarMessageRetrievalTopic">
               <last-value-queue>true</last-value-queue>
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
              <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <max-size-bytes>26214400</max-size-bytes> 
               <address-full-policy>DROP</address-full-policy>
          </address-setting>
          
          <address-setting match="jms.queue.toolbarCommandQueue">
               <last-value-queue>true</last-value-queue>
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
              <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <max-size-bytes>26214400</max-size-bytes> 
               <address-full-policy>DROP</address-full-policy>
          </address-setting>
          
          <address-setting match="jms.queue.DLQ">
               <max-size-bytes>10485760</max-size-bytes>
               <address-full-policy>DROP</address-full-policy>
          </address-setting>
          
          <address-setting match="jms.queue.ExpiryQueue">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
               <max-size-bytes>10485760</max-size-bytes> 
               <address-full-policy>DROP</address-full-policy>
          </address-setting>
             
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>26214400</max-size-bytes>
         <page-size-bytes>10485760</page-size-bytes>       
         <message-counter-history-day-limit>30</message-counter-history-day-limit>
         <address-full-policy>PAGE</address-full-policy>
      </address-setting>
   </address-settings>

</configuration>

And my hornetq jms configuration:

<configuration xmlns="urn:hornetq"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">
     
     <connection-factory name="NettyConnectionFactory">
        <connectors>
         <connector-ref connector-name="netty"/>
        </connectors>
          <entries>
               <entry name="java:/ConnectionFactory"/>
               <entry name="/XAConnectionFactory"/>
          </entries>
          <retry-interval>2000</retry-interval>
          <retry-interval-multiplier>1</retry-interval-multiplier>
          <max-retry-interval>2000</max-retry-interval>
          <reconnect-attempts>1000</reconnect-attempts>
     </connection-factory>
     
     <connection-factory name="NettyThroughputConnectionFactory">
        <connectors>
         <connector-ref connector-name="netty-throughput"/>
        </connectors>
          <entries>
               <entry name="java:/ThroughputConnectionFactory"/>
               <entry name="/XAThroughputConnectionFactory"/>
          </entries>
          <retry-interval>2000</retry-interval>
          <retry-interval-multiplier>1</retry-interval-multiplier>
          <max-retry-interval>2000</max-retry-interval>
          <reconnect-attempts>1000</reconnect-attempts>
     </connection-factory>
     
     <!-- DEFAULT QUEUES -->
     <queue name="DLQ">
          <entry name="/queue/DLQ"/>
     </queue>
     <queue name="ExpiryQueue">
          <entry name="/queue/ExpiryQueue"/>
     </queue>   
     
     <!-- Toolbar Event Logging Queue -->
        <queue name="toolbarEventQueue">
          <entry name="/queue/toolbarEventQueue" />
     </queue>
     
     <!-- Toolbar Command Queue -->
        <queue name="toolbarCommandQueue">
          <entry name="/queue/toolbarCommandQueue" />
     </queue>
     
     <!-- Toolbar Message Retrieval Topic -->
        <topic name="toolbarMessageRetrievalTopic">
          <entry name="/topic/toolbarMessageRetrievalTopic" />
     </topic>

</configuration>

1. Re: Help with unexpected HornetQ crash.

unsavory Jul 8, 2010 3:43 PM (in response to unsavory)

Some other exceptions:
javax.jms.JMSException: Timed out waiting for response when sending packet 71
       at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
       at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:285)
       at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
       at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)
       at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:199)
...

and...
javax.jms.JMSException: Timed out waiting for response when sending packet 43
       at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:277)
       at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:514)
       at org.hornetq.core.client.impl.DelegatingSession.commit(DelegatingSession.java:156)
       at org.hornetq.jms.client.HornetQMessage.acknowledge(HornetQMessage.java:857)

It seems that the HornetQ server itself crashed since all the producers and consumers seem to be timing out waiting for a response.
Actions
2. Re: Help with unexpected HornetQ crash.

igorhvr Dec 27, 2010 2:50 PM (in response to unsavory)

Your best bet is to increase the callTimeout in the ClientSessionFactory you use. If the server is just temporarily overloaded this will give your client sessions more time before giving up and throwing an exception (which will make things even worse in the server side).

I had this and similar exceptions fairly frequently (a couple of times every day - however when it happened the error bubbled up and crashed the application) in my production setup - after increasing my timeout to five minutes (the default is 30 seconds), I never saw any errors like this again.
Actions

Go to original post