Hornetq Issue during Stress test
All,
We are trying to replace our messaging provider with HornetQ. As of today, I'm running HornetQ in standalone mode (Paging enabled) and all our functional tests gives good result. However when i try to do a stree test with fast producer and slow consumer in the console i'm seeing following exception,
[Old I/O server worker (parentId: 1493422662, [id: 0x5903d246, /localhost:5445])] 23:58:14,457 SEVERE [org.hornetq.core.protocol.core.ServerSessionPacketHandler]
Caught unexpected exception
java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.jboss.netty.buffer.HeapChannelBuffer.getBytes(HeapChannelBuffer.java:107)
at org.jboss.netty.buffer.DynamicChannelBuffer.getBytes(DynamicChannelBuffer.java:138)
at org.jboss.netty.buffer.HeapChannelBuffer.setBytes(HeapChannelBuffer.java:132)
at org.jboss.netty.buffer.AbstractChannelBuffer.writeBytes(AbstractChannelBuffer.java:462)
at org.hornetq.core.buffers.impl.ChannelBufferWrapper.writeBytes(ChannelBufferWrapper.java:572)
at org.hornetq.core.message.impl.MessageImpl.encode(MessageImpl.java:453)
at org.hornetq.core.paging.impl.PagedMessageImpl.encode(PagedMessageImpl.java:147)
at org.hornetq.core.paging.impl.PageImpl.write(PageImpl.java:188)
at org.hornetq.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:878)
at org.hornetq.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:320)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:886)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:634)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:561)
at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1365)
at org.hornetq.core.server.impl.ServerSessionImpl.sendContinuations(ServerSessionImpl.java:1137)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:457)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)
at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)
at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:514)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:127)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:357)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
It apears to me first that it is a FATAL error, however the server still seem to be consuming and delivering messages to my producers and consumers. The biggest concern that we have around promoting this to production environments is the queue conut that we see. It shows up in negative value after some time. I'm not sure from when it shows up in negative numbers.
To debug this further i have changed my paging configuration to something like below,
<address-settings>
<!--default for catch all-->
<!--TODO: Muthu to redefine it after checking message arriaval rate for each address -->
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<!--
<page-size-bytes>10485760</page-size-bytes>
<max-size-bytes>104857600</max-size-bytes>
-->
<page-size-bytes>1</page-size-bytes>
<max-size-bytes>2</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
This allowed me to see all my messages in the disk during the test and i can see more messages in the paged directory (in multiple lakhs). However the queue count through the following API call shows up the numbers in negative...(say -80817) and the count seem to be wrong also.
I'm using the following API call to get the count
JMSQueueControl queueControl = (JMSQueueControl)MBeanServerInvocationHandler.newProxyInstance(mbsc, on, JMSQueueControl.class, false);
Should i worry about ArrayOutOfBoundException - Can someone please explain?
How to get the actual queue count correctly all the time ?
Note: The entire testing is happening in Linux Environment.
