Large Messages are not handled properly in Hornetq Clustering
hushen.savani Dec 27, 2011 9:46 AMHi,
I have my application running on JBossAS-6.1.1 with Hornetq-2.2.5. I have clustered my application (which is of cource written with Honetq apis (JMS)). I have tested my application with clustering (2 nodes) with normal sized messages. It works fine. It load balances the messages between two nodes in RR Fashion properly. But when transmitting the large messages, following exception is thrown in one of the nodes:
19:28:33,251 WARN [org.hornetq.core.server.impl.QueueImpl] removing consumer which did not handle a message, consumer=org.hornetq.core.server.cluster.impl.ClusterConnectionBridge@66b2cd4f, message=Reference[3679]:RELIABLE:ServerMessage[messageID=3679,priority=4,expiration=[Tue Dec 27 19:28:43 IST 2011], durable=true, address=jms.queue.mocmMasterQueue,properties=TypedProperties[{_HQ_LARGE_SIZE=271064, _HQ_ROUTE_TOsf.mocm-cluster.afdcd7f8-3092-11e1-8e63-002128bc0745=[B@2abc60d6}]]: java.lang.NullPointerException at org.hornetq.core.buffers.impl.ResetLimitWrappedHornetQBuffer.<init>(ResetLimitWrappedHornetQBuffer.java:39) [:] at org.hornetq.core.message.impl.MessageImpl.getBodyBuffer(MessageImpl.java:267) [:] at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:223) [:] at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142) [:] at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:407) [:] at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2017) [:] at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1587) [:] at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:] at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:] at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_12] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_12] at java.lang.Thread.run(Thread.java:619) [:1.6.0_12] 19:28:33,259 ERROR [org.hornetq.utils.OrderedExecutorFactory] Caught unexpected Throwable: java.util.NoSuchElementException at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:189) [:] at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1604) [:] at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:] at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:] at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_12] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_12] at java.lang.Thread.run(Thread.java:619) [:1.6.0_12] 19:29:12,898 ERROR [org.hornetq.core.postoffice.impl.PostOfficeImpl] failed to expire messages for queue sf.mocm-cluster.afdcd7f8-3092-11e1-8e63-002128bc0745: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793) [:1.6.0_12] at java.util.HashMap$KeyIterator.next(HashMap.java:828) [:1.6.0_12] at org.hornetq.core.postoffice.impl.PostOfficeImpl.cleanupInternalPropertiesBeforeRouting(PostOfficeImpl.java:815) [:] at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:591) [:] at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:556) [:] at org.hornetq.core.server.impl.QueueImpl.move(QueueImpl.java:1907) [:] at org.hornetq.core.server.impl.QueueImpl.expire(QueueImpl.java:875) [:] at org.hornetq.core.server.impl.QueueImpl.expireReferences(QueueImpl.java:1098) [:] at org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper.run(PostOfficeImpl.java:1240) [:] at java.lang.Thread.run(Thread.java:619) [:1.6.0_12]
I have explored that similer bug is reolved in JBoss-5.1.1-EAP. Can anyone please suggest pointers on how to resolve this issue with JBossAS-6.1.1?
Thank you.