6 Replies Latest reply on Nov 29, 2013 1:20 AM by patilpr13

    HornetQ 2.2.14 - Blocked messages in queue

    julien.victor
    julien.victor Sep 24, 2012 12:52 PM

      Hello

       

      We are using HornetQ in one of our environment and found out that messages are stuck in one of our queue.

       

      Environment detail :

      • JBoss 5.1.0 EAP
      • HornetQ 2.2.14 Community
      • 2 Front Office instances (with producers, and a store and forward configuration (target = middle office instance))
      • 1 dedicated Middle Office JBoss instance (with HornetQ)
      • 1 BackOffice Office JBoss instance (with consumers)

      Note: All instances are restarted weekly.

       

      Issue:

      We detected that a few days ago, messages were beginning to be stuck in a queue in our middle office instance. Front Offices were continiously creating messages, and Back Office instance did not detect any new message to process for that queue (other queues were behaviouring correctly).

       

      At the end of the day, there were 110k messages blocked.

       

      At night, we made instance restart (for Middle Office and then Back Office). The majority of messages were then proceeded, thought 33k would stay stuck in the Middle Office.

       

      We restarted two more time our instances but we still have 33k (according to JMX).

       

      Copying the data folder in another JBoss instance doesn't help as the message counter still is 33k, but messages are still not detected.

       

      What we'ld like to know:

      First of all, are blocked messages a known issue (unfortunatly, I can retreive JBoss log files) ?

      Then, is is possible to extract messages from the HornetQ journal, without a running HornetQ instance ? Can they even be retrieve ?

      For you information, here is the following trace trace at JBoss startup.

      2012-09-24 01:51:43,146 ERROR [org.hornetq.core.protocol.core.ServerSessionPacketHandler] (Old I/O server worker (parentId: 2010453219, [id: 0x77d514e3, /0.0.0.0:64436])) : Caught exception
HornetQException[errorCode=104 message=large-message not initialized on server]
    at org.hornetq.core.server.impl.ServerSessionImpl.sendContinuations(ServerSessionImpl.java:1175)
    at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:458)
    at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:508)
    at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:556)
    at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:517)
    at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:533)
    at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)
    at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
    at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372)
    at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:534)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)
    at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:127)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
    at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:372)
    at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:367)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
    at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
    at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

      To me, large-message configuration is set (see hornetq-configuration.xml):

         <large-messages-directory>${jboss.server.data.dir}/hornetq/largemessages</large-messages-directory>

       

      Thank you for any idea/help you could provide

      • 2023 Views
      • Tags: