Management API moveMessages does not work with Large Messages

[r.reimann]

While porting to JBoss AS6 our JMS client which moves (by consuming + sending without explicit modifications via JMS API) messages from queue A to queue B stopped working due to https://issues.jboss.org/browse/HORNETQ-538 since we are using ObjectMessage and HornetQ seems to try implicit modifications before sending which lead to the "This is a read-only buffer, setOperations are not supported" error.

To workaround this problem i changed our JMS client to use the HornetQ Management API (method moveMessages) with cache-large-message-client property enabled but this leads to a similar kind of problem. The moveMessages operation fails if there is a server restart inbetween throwing:

08:16:29,835 ERROR [org.hornetq.core.server.impl.ServerConsumerImpl] Log4jLogDelegate Failed to run large message deliverer: java.lang.NegativeArraySizeException

08:16:29,835 ERROR [org.hornetq.core.server.impl.ServerConsumerImpl] Log4jLogDelegate Failed to run large message deliverer: java.lang.NegativeArraySizeException
08:16:29,860 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] Log4jLogDelegate Failed to deliver message: java.lang.RuntimeException
     at org.hornetq.core.client.impl.LargeMessageBufferImpl.resetReaderIndex(LargeMessageBufferImpl.java:716) [:6.0.0.Final]
08:16:29,910 ERROR [org.hornetq.utils.OrderedExecutorFactory] Log4jLogDelegate Caught unexpected Throwable: java.lang.IllegalAccessError: LargeMessage have read-only and one-way buffers

Thereby it doesn't matter if i try to use to move a message by using our own client or the HornetQ MBean inside the jmx-console.

Does HORNETQ-538 also cover the latter error and is there already a schedule for the HornetQ 2.2.0 release? Are there any workarounds to get the moveMessages operations working with large messages?

Any help is appreciated.

Regards

Robert

[clebert.suconic]

This should have been fixed by trunk now.

Do you think you could verify that?

[r.reimann]

I could try to verify that if you could provide some instructions on how to update HornetQ inside AS6. Which hornetq jars, config-files and other dependencies do i have to replace? Trying to figure that out manually could get a real pain and leave me with a corrupted installation. An (Ant) build script like the one for the AS5 integration would be a great help to provide a consistent update mechanism.

Do you provide snapshot releases of the distribution in your maven repository or do i have to build the trunk from source?

[r.reimann]

The AS5 script config/jboss-as-5/build.xml seems to work for AS6 so i don't need further update instructions. But it would be nice if you could include a dedicated AS6 script and update instructions for AS6 in the documentation for the next release.

Btw. while building the trunk from source i got a build failure which i could workaround by commenting out the org.slf4j.lib fileset in build-hornetq.xml:

bin-distro:
    [mkdir] Created dir: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build/hornetq-2.2.0.CR1
    [mkdir] Created dir: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build/hornetq-2.2.0.CR1/lib
    [mkdir] Created dir: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build/hornetq-2.2.0.CR1/config
    [mkdir] Created dir: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build/hornetq-2.2.0.CR1/bin
    [mkdir] Created dir: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build/hornetq-2.2.0.CR1/docs/api
    [mkdir] Created dir: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build/hornetq-2.2.0.CR1/licenses


BUILD FAILED
/Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build.xml:211: The following error occurred while executing this line:
/Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/build-hornetq.xml:1230: /Users/rr/Downloads/_jboss6-src/hornetq-trunk/trunk/${org.slf4j.lib} not found.

[r.reimann]

Hi Clebert,

i could verified, that the issues are fixed in the trunk. So can you tell me when we can expect a 2.2.0.Final or at least a 2.2.CR1? I can hardly convince our operations department to take a self-compiled trunk version into production.

Contrary to my previous comment the AS5 script didn't work well for AS6. The configuration files are different and the resource adapter name changed from hornetq-ra to jms-ra in AS6. So i left the configuration unchanged and just replaced all jars that matched the filesets in the AS5 script.

Apart from that i had further issues during the update. I had to delete all hornetq related data (bindings, journal, largemessages) after the update to get my server running again. Otherwise i got the exceptions below during startup:

15:30:18,550 SCHWERWIEGEND [HornetQServerImpl] Failure in initialisation: java.lang.IllegalStateException: Invalid record type 23
        at org.hornetq.core.persistence.impl.journal.JournalStorageManager.loadBindingJournal(JournalStorageManager.java:1367) [:]

15:52:21,077 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] Log4jLogDelegate Failed to deliver message: javax.ejb.EJBTransactionRolledbackException: org.hornetq.jms.client.HornetQMessage cannot be cast to javax.jms.TextMessage

Regards

Robert

[clebert.suconic]

I don't think you can get this into production yet. I asked you to verify only...

We are working on testing now. So, it should be some time by the end of next month.

The demand I have is for earlier than end of Feb...     but software development sometimes is unpredictable. We are working hard to make it happen sooner than late Feb.

[r.reimann]

hank you for sharing the planed schedule.

I'm sorry but i have got bad news concerning the movement of large messages. During my testing today i got "java.lang.IllegalAccessError: LargeMessage have read-only and one-way buffers" when trying to move messages after a server restart. See the stacktrace for the current exceptions below.

So i reviewed my positive results from yesterday an noticed, that all the messages i picked during the positive tests were below "min-large-message-size" and therefore not valid for testing my issue.

After the error the messages vanish entirely and are not even present in the DLQ.

Should i create a Jira Issue on that or do you want to perform your own verification attempts before?

10:13:28,412 ERROR [org.hornetq.core.server.impl.ServerConsumerImpl] Log4jLogDelegate Failed to run large message deliverer: java.lang.NegativeArraySizeException
          at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:47) [:]
          at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:39) [:]
          at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:139) [:]
          at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:126) [:]
          at org.hornetq.api.core.HornetQBuffers.fixedBuffer(HornetQBuffers.java:91) [:]
          at org.hornetq.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:822) [:]
          at org.hornetq.core.server.impl.ServerConsumerImpl$1.run(ServerConsumerImpl.java:717) [:]
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_22]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_22]
          at java.lang.Thread.run(Thread.java:680) [:1.6.0_22]


10:13:28,418 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] Log4jLogDelegate Failed to deliver message: java.lang.IndexOutOfBoundsException
          at org.jboss.netty.buffer.AbstractChannelBuffer.setIndex(AbstractChannelBuffer.java:67) [:]
          at org.hornetq.core.buffers.impl.ChannelBufferWrapper.setIndex(ChannelBufferWrapper.java:497) [:]
          at org.hornetq.core.message.impl.MessageImpl.createBody(MessageImpl.java:942) [:]
          at org.hornetq.core.client.impl.ClientLargeMessageImpl.checkBuffer(ClientLargeMessageImpl.java:189) [:]
          at org.hornetq.core.client.impl.ClientLargeMessageImpl.getBodyBuffer(ClientLargeMessageImpl.java:100) [:]
          at org.hornetq.jms.client.HornetQMessage.doBeforeReceive(HornetQMessage.java:890) [:]
          at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:264) [:]
          at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:841) [:]
          at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:45) [:]
          at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:958) [:]
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_22]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_22]
          at java.lang.Thread.run(Thread.java:680) [:1.6.0_22]


10:13:28,429 WARN  [org.hornetq.core.server.impl.QueueImpl] Log4jLogDelegate Message has reached maximum delivery attempts, sending it to Dead Letter Address jms.queue.DLQ from jms.queue.RatingAktenQueue
10:13:28,431 ERROR [org.hornetq.core.server.impl.ServerConsumerImpl] Log4jLogDelegate Failed to run large message deliverer: java.lang.NegativeArraySizeException
          at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:47) [:]
          at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:39) [:]
          at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:139) [:]
          at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:126) [:]
          at org.hornetq.api.core.HornetQBuffers.fixedBuffer(HornetQBuffers.java:91) [:]
          at org.hornetq.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer.deliver(ServerConsumerImpl.java:822) [:]
          at org.hornetq.core.server.impl.ServerConsumerImpl$1.run(ServerConsumerImpl.java:717) [:]
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_22]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_22]
          at java.lang.Thread.run(Thread.java:680) [:1.6.0_22]


10:13:28,432 ERROR [org.hornetq.utils.OrderedExecutorFactory] Log4jLogDelegate Caught unexpected Throwable: java.lang.IllegalAccessError: LargeMessage have read-only and one-way buffers
          at org.hornetq.core.client.impl.LargeMessageControllerImpl.checkForPacket(LargeMessageControllerImpl.java:1338) [:]
          at org.hornetq.core.client.impl.LargeMessageControllerImpl.discardUnusedPackets(LargeMessageControllerImpl.java:142) [:]
          at org.hornetq.core.client.impl.ClientLargeMessageImpl.discardBody(ClientLargeMessageImpl.java:169) [:]
          at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:850) [:]
          at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:45) [:]
          at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:958) [:]
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_22]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_22]
          at java.lang.Thread.run(Thread.java:680) [:1.6.0_22]

[clebert.suconic]

Is this on trunk?

If yes, can you raise a JIRA with a simple testcase?

[r.reimann]

Yes, verified today on current trunk version (rev. 10128) and created https://issues.jboss.org/browse/HORNETQ-627 and attached a testcase.