1 2 Previous Next 16 Replies Latest reply on Jan 16, 2012 4:35 AM by Yong Hao Gao

    Large Messages are not handled properly in Hornetq Clustering

    Hushen Savani Novice

      Hi,

          

           I have my application running on JBossAS-6.1.1 with Hornetq-2.2.5. I have clustered my application (which is of cource written with Honetq apis (JMS)).  I have tested my application with clustering (2 nodes) with normal sized messages. It works fine. It load balances the messages between two nodes in RR Fashion properly. But when transmitting the large messages, following exception is thrown in one of the nodes:

       

      19:28:33,251 WARN  [org.hornetq.core.server.impl.QueueImpl] removing consumer which did not handle a message, consumer=org.hornetq.core.server.cluster.impl.ClusterConnectionBridge@66b2cd4f, message=Reference[3679]:RELIABLE:ServerMessage[messageID=3679,priority=4,expiration=[Tue Dec 27 19:28:43 IST 2011], durable=true, address=jms.queue.mocmMasterQueue,properties=TypedProperties[{_HQ_LARGE_SIZE=271064, _HQ_ROUTE_TOsf.mocm-cluster.afdcd7f8-3092-11e1-8e63-002128bc0745=[B@2abc60d6}]]: java.lang.NullPointerException
                at org.hornetq.core.buffers.impl.ResetLimitWrappedHornetQBuffer.<init>(ResetLimitWrappedHornetQBuffer.java:39) [:]
                at org.hornetq.core.message.impl.MessageImpl.getBodyBuffer(MessageImpl.java:267) [:]
                at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:223) [:]
                at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142) [:]
                at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:407) [:]
                at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2017) [:]
                at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1587) [:]
                at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:]
                at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:]
                at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:]
                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_12]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_12]
                at java.lang.Thread.run(Thread.java:619) [:1.6.0_12]
      
      
      19:28:33,259 ERROR [org.hornetq.utils.OrderedExecutorFactory] Caught unexpected Throwable: java.util.NoSuchElementException
                at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:189) [:]
                at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1604) [:]
                at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:]
                at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:]
                at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:]
                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_12]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_12]
                at java.lang.Thread.run(Thread.java:619) [:1.6.0_12]
      
      
      19:29:12,898 ERROR [org.hornetq.core.postoffice.impl.PostOfficeImpl] failed to expire messages for queue sf.mocm-cluster.afdcd7f8-3092-11e1-8e63-002128bc0745: java.util.ConcurrentModificationException
                at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) [:1.6.0_12]
                at java.util.HashMap$KeyIterator.next(HashMap.java:828) [:1.6.0_12]
                at org.hornetq.core.postoffice.impl.PostOfficeImpl.cleanupInternalPropertiesBeforeRouting(PostOfficeImpl.java:815) [:]
                at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:591) [:]
                at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:556) [:]
                at org.hornetq.core.server.impl.QueueImpl.move(QueueImpl.java:1907) [:]
                at org.hornetq.core.server.impl.QueueImpl.expire(QueueImpl.java:875) [:]
                at org.hornetq.core.server.impl.QueueImpl.expireReferences(QueueImpl.java:1098) [:]
                at org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper.run(PostOfficeImpl.java:1240) [:]
                at java.lang.Thread.run(Thread.java:619) [:1.6.0_12]
      

       

      I have explored that similer bug is reolved in JBoss-5.1.1-EAP. Can anyone please suggest pointers on how to resolve this issue with JBossAS-6.1.1?

       

      Thank you.

        1 2 Previous Next