6 Replies Latest reply on Feb 18, 2016 11:38 AM by jbertram

HQ224045: Failed to run large message deliverer

arnab_ghosh Feb 17, 2016 5:31 PM

Hi All,

We are using Wildfly 8.2.0.Final. I seeing the below exceptions on the Wildfly instance which deploys the JMS Queues.

2016-02-17 17:23:50,963 ERROR [org.hornetq.core.server] (Thread-22 (HornetQ-server-HornetQServerImpl::serverUUID=fa9f3bb2-bb08-11e5-b71a-e7c40d5c561c-673764043)) HQ224045: Failed to run large message deliverer: java.lang.UnsupportedOperationException

at java.nio.ByteBuffer.array(ByteBuffer.java:994) [rt.jar:1.8.0_66]

at org.hornetq.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1097) [hornetq-server-2.4.5.Final.jar:]

at org.hornetq.core.server.impl.ServerConsumerImpl$2.run(ServerConsumerImpl.java:976) [hornetq-server-2.4.5.Final.jar:]

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]

2016-02-17 17:23:50,965 ERROR [org.hornetq.core.server] (Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=fa9f3bb2-bb08-11e5-b71a-e7c40d5c561c-673764043)) HQ224045: Failed to run large message deliverer: java.lang.UnsupportedOperationException

at java.nio.ByteBuffer.array(ByteBuffer.java:994) [rt.jar:1.8.0_66]

at org.hornetq.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:1097) [hornetq-server-2.4.5.Final.jar:]

at org.hornetq.core.server.impl.ServerConsumerImpl$2.run(ServerConsumerImpl.java:976) [hornetq-server-2.4.5.Final.jar:]

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.5.Final.jar:]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_66]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_66]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_66]

There are almost 2000 messages stuck in the queue which are not getting delivered.

In the largemessages directory I am seeing a few messages with 0 size. I tried deleting those files and restart but no the .msg files got recreated.

Please help.

Regards

Arnab

1. Re: HQ224045: Failed to run large message deliverer

arnab_ghosh Feb 17, 2016 5:33 PM (in response to arnab_ghosh)

We are using SAN storage for hornetQ persistence and jre 1.8.0_66. Please let me know if more information is required.
Actions
2. Re: HQ224045: Failed to run large message deliverer

jbertram Feb 17, 2016 5:48 PM (in response to arnab_ghosh)

One key question is why do you have "large" messages of size 0. Do you have clients publishing "large" messages with no actual body?

FYI - This was addressed upstream in Artemis on https://issues.apache.org/jira/browse/ARTEMIS-331.
Actions
3. Re: HQ224045: Failed to run large message deliverer

arnab_ghosh Feb 17, 2016 6:04 PM (in response to jbertram)

Justin,

Clients are sending object messages I will definitely verify if they are ever sending null.
If the body is indeed null, how is the message getting classified as a large message ? Is it not based on the message size ?

My address settings is as below:

                    <address-setting match="#">
                         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <redelivery-delay>5000</redelivery-delay>
                        <max-size-bytes>10485760</max-size-bytes>
                        <page-size-bytes>2097152</page-size-bytes>
                        <address-full-policy>PAGE</address-full-policy>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                        <max-delivery-attempts>10</max-delivery-attempts>
                    </address-setting>
Actions
4. Re: HQ224045: Failed to run large message deliverer

arnab_ghosh Feb 17, 2016 6:26 PM (in response to arnab_ghosh)

is there a way to just delete these null messages so the others go through ? I tried stopping all the servers and removing .msg file which shows size as zero but it did not work. The files got re-created.
Actions
5. Re: HQ224045: Failed to run large message deliverer

jbertram Feb 18, 2016 11:36 AM (in response to arnab_ghosh)

Messaging can be detected by the server as large and treated as such or a client may force them to be large. You need to determine what exactly the clients are doing.
Actions
6. Re: HQ224045: Failed to run large message deliverer

jbertram Feb 18, 2016 11:38 AM (in response to arnab_ghosh)

Ostensibly, you can shut down the server, remove the large messages, and restart the server. However, if clients continue to send large messages with a zero-length body then they'll just come back. I'm assuming here that the zero-length messages are coming directly from clients and are not being caused by some other kind of bug.
Actions

Go to original post