1 Reply Latest reply on Nov 8, 2012 11:46 AM by manu_1185

    Getting timeout on server and client

    manu_1185

      Hi,

       

      We are using hornetq for our chat application on mobile phones. Our application creates a consumer (with message selectors) queues for every user. We constantly create and close consumers as users connect and disconnect. Under a load of around 3000-4000 clients, we start getting timeout exceptions on hornetq server and our application (during consumer creation and sending message acknowledgements). We have already made message sends non-blocking by setting block-on-durable-send to false. It has become a blocker for us right now and we are not able to scale. Any ideas what might be wrong. Attached is the configuration file and given below are the exceptions:

       

      [Old I/O server worker (parentId: 12874272, [id: 0x00c47220, /10.240.0.46:5445])] 8-Nov 10:54:57,942 WARNING [QueueImpl]  Timed out on waiting for MessageCount

      java.lang.IllegalStateException: Timed out on waiting for MessageCount

              at org.hornetq.core.server.impl.QueueImpl.getMessageCount(QueueImpl.java:828)

              at org.hornetq.core.server.impl.ServerSessionImpl.executeQueueQuery(ServerSessionImpl.java:526)

              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:222)

              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:517)

              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:533)

              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)

              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)

              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372)

              at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:534)

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)

              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)

              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)

              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372)

              at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:367)

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)

              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)

              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)

              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)

              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

       

      javax.jms.JMSException: Timed out waiting for response when sending packet 43

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

              at org.hornetq.core.client.impl.ClientSessionImpl.commit(ClientSessionImpl.java:528)

              at org.hornetq.core.client.impl.DelegatingSession.commit(DelegatingSession.java:157)

              at org.hornetq.jms.client.HornetQMessage.acknowledge(HornetQMessage.java:857)

              at com.bsb.hike.pubsub.jms.JMSConsumer$JMSMessageListener.acknowledge(JMSConsumer.java:286)

              at com.bsb.hike.im.IMMessageHandler.handlePublishAck(IMMessageHandler.java:677)

              at com.bsb.hike.mqtt.MqttMessageHandler.handleMessage(MqttMessageHandler.java:314)

              at com.bsb.hike.mqtt.MqttMessageHandler.handleMessage(MqttMessageHandler.java:204)

              at com.bsb.hike.mqtt.MqttMessageHandler.messageReceived(MqttMessageHandler.java:158)

              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)

              at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:43)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:129)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)

              at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:270)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)

              at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.run(ChannelUpstreamEventRunnable.java:44)

              at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:315)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

              at java.lang.Thread.run(Thread.java:722)

       

      javax.jms.JMSException: Timed out waiting for response when sending packet 30

              at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:695)

              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:264)

              at org.hornetq.jms.client.HornetQConnection.createSessionInternal(HornetQConnection.java:521)

              at org.hornetq.jms.client.HornetQConnection.createSession(HornetQConnection.java:161)

              at com.bsb.hike.pubsub.jms.JMSConnection.createSession(JMSConnection.java:51)

              at com.bsb.hike.pubsub.jms.JMSConsumer.<init>(JMSConsumer.java:76)

              at com.bsb.hike.pubsub.jms.JMSPubSubManager.receive(JMSPubSubManager.java:241)

              at com.bsb.hike.mqtt.MqttMessageHandler.handleMessage(MqttMessageHandler.java:417)

              at com.bsb.hike.mqtt.MqttMessageHandler.handleMessage(MqttMessageHandler.java:210)

              at com.bsb.hike.mqtt.MqttMessageHandler.messageReceived(MqttMessageHandler.java:158)

              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:95)

              at org.jboss.netty.handler.timeout.IdleStateAwareChannelHandler.handleUpstream(IdleStateAwareChannelHandler.java:43)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.messageReceived(SimpleChannelUpstreamHandler.java:129)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)

              at org.jboss.netty.handler.timeout.IdleStateHandler.messageReceived(IdleStateHandler.java:270)

              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:75)

              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)

              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:792)

              at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.run(ChannelUpstreamEventRunnable.java:44)

              at org.jboss.netty.handler.execution.OrderedMemoryAwareThreadPoolExecutor$ChildExecutor.run(OrderedMemoryAwareThreadPoolExecutor.java:315)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

              at java.lang.Thread.run(Thread.java:722)

        • 1. Re: Getting timeout on server and client
          manu_1185

          Also getting this when closing consumers. Given below is the exception.

           

          Thu Nov 08 14:26:53 2012 com.bsb.hike.pubsub.jms.JMSConsumer:? ERROR Thread-3939 (group:HornetQ-client-global-threads-1189682735) Error recovering message consumer : HornetQMessageConsumer[org.hornetq.core.client.impl.ClientConsumerImpl@7e62f8bc]. Error : Timed out waiting for response when sending packet 44

          javax.jms.JMSException: Timed out waiting for response when sending packet 44

                  at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:276)

                  at org.hornetq.core.client.impl.ClientSessionImpl.rollback(ClientSessionImpl.java:583)

                  at org.hornetq.core.client.impl.DelegatingSession.rollback(DelegatingSession.java:477)

                  at org.hornetq.jms.client.HornetQSession.recover(HornetQSession.java:294)

                  at com.bsb.hike.pubsub.jms.JMSConsumer.close(JMSConsumer.java:174)

                  at com.bsb.hike.pubsub.jms.JMSPubSubManager.stopReceiving(JMSPubSubManager.java:299)

                  at com.bsb.hike.mqtt.MqttClientSession.close(MqttClientSession.java:136)

                  at com.bsb.hike.mqtt.MqttClientManager.unregister(MqttClientManager.java:171)

                  at com.bsb.hike.im.IMMessageHandler.onMessage(IMMessageHandler.java:698)

                  at com.bsb.hike.pubsub.jms.JMSConsumer$JMSMessageListener.onMessage(JMSConsumer.java:262)

                  at org.hornetq.jms.client.JMSMessageListenerWrapper.onMessage(JMSMessageListenerWrapper.java:91)

                  at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:866)

                  at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:44)

                  at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:983)

                  at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                  at java.lang.Thread.run(Thread.java:722)