Consumer not reachable or overwhelmed?

[khanna111]

Hi,

While testing out with 5 publisher threads, I get this on the hornetq primary server (there is a backup server as well). It seems that I can publish messages but the consumer seems to have crashed but am not sure about the last part since the consumer is running as part of a tomcat vm.

This is the exception on the hornetq server: (exception on the tomcat server is given after that)

java.lang.IllegalStateException: 31639999 Could not find reference on consumerID=0, messageId = 2148064229 queue = jms.queue.exampleQueue closed = false

        at org.hornetq.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:524)

        at org.hornetq.core.server.impl.ServerSessionImpl.handleAcknowledge(ServerSessionImpl.java:648)

        at org.hornetq.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:157)

        at org.hornetq.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:466)

        at org.hornetq.core.remoting.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:423)

        at org.hornetq.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:389)

        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:432)

        at org.hornetq.integration.transports.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:67)

        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:98)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:358)

        at org.hornetq.integration.transports.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:171)

        at org.hornetq.integration.transports.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:136)

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)

"hornetq.log" 49357 lines, 4779489 characters

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:345)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:332)

        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)

        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:275)

        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:196)

        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

        at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:179)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)

        at java.lang.Thread.run(Thread.java:595)

[New I/O server worker #1-1] 16:23:40,052 SEVERE [org.hornetq.core.server.impl.ServerSessionImpl]  Failed to acknowledge

java.lang.IllegalStateException: 31639999 Could not find reference on consumerID=0, messageId = 2147821298 queue = jms.queue.exampleQueue closed = false

        at org.hornetq.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:524)

        at org.hornetq.core.server.impl.ServerSessionImpl.handleAcknowledge(ServerSessionImpl.java:648)

        at org.hornetq.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:157)

        at org.hornetq.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:466)

        at org.hornetq.core.remoting.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:423)

        at org.hornetq.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:389)

        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:432)

        at org.hornetq.integration.transports.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:67)

        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:98)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)

        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:796)

        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:358)

        at org.hornetq.integration.transports.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:171)

        at org.hornetq.integration.transports.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:136)

        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)

        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)

------------------------------------------------------------------

Exception on the tomcat server:

javax.jms.JMSException: Timed out waiting for response when sending packet 71
        at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:270)
        at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:269)
        at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
        at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:455)
        at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:208)
        at main.Main.doGet(Main.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:619)
Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71]
        ... 20 more
javax.jms.JMSException: Timed out waiting for response when sending packet 71
        at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:270)
        at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:269)
        at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
        at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:455)
        at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:208)
        at main.Main.doGet(Main.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:619)
Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71]
        ... 20 more
javax.jms.JMSException: Timed out waiting for response when sending packet 71
        at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:270)
        at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:269)
        at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
        at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:455)
        at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:208)
        at main.Main.doGet(Main.java:103)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:619)
Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 71]
        ... 20 more

Attachments:

The consumer code is in JMSHelper.java

The publisher code is in Main.java

[timfox]

Try TRUNK.

This has been discussed several times in the last few weeks on this forum, there's also a JIRA for it if you care to look.

[khanna111]

Thanks for the update.

Seems as if the bug no: HORNETQ-275 and the fix version is 2.1.0 beta2.

We might not want to go into production with a beta release. When is the production release 2.1.0 GA estimated?

Thanks

Gaurav

[clebert.suconic]

I would say calling trunk GA is just a matter of naming it.

You can follow the progress towards CR1 here:

https://jira.jboss.org/jira/secure/IssueNavigator.jspa?reset=true&pid=12310830&fixfor=12314884

As long as you test your system.. you should be pretty safe at this point.