8 Replies Latest reply on Aug 16, 2017 11:13 AM by jbertram

[WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 15, 2017 5:17 AM

Hi experts,

I am facing the problem that messages are not forwarded between 2 cluster nodes. I check the server.log and below exceptions are seen on 1 server. It happen 2 times for 2 different cluster nodes which are connected to current server:

2017-08-02 17:06:46,096 WARN [Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ222151: removing consumer which did not handle a message, consumer=ClusterConnectionBridge@fff7f6e [name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, queue=QueueImpl[name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1ce980b3 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@fff7f6e [name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, queue=QueueImpl[name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1ce980b3 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?key-store-password=****&batch-delay=200&port=5455&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****e&tcp-receive-buffer-size=200000&nio-remoting-threads=5&use-nio=true], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1086091769[nodeUUID=f9723119-e8f9-11e6-a77b-3d7585531af6, connector=TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true, address=jms, server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6])) [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?key-store-password=****&batch-delay=200&port=5455&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****&tcp-receive-buffer-size=200000&nio-remoting-threads=5&use-nio=true], discoveryGroupConfiguration=null]], message=Reference[34376129663]:NON-RELIABLE:ServerMessage[...]@498949710: java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 5348 (expected: 0 <= readerIndex <= writerIndex <= capacity(17))

at io.netty.buffer.AbstractByteBuf.setIndex(AbstractByteBuf.java:99)

at io.netty.buffer.WrappedByteBuf.setIndex(WrappedByteBuf.java:116)

at org.hornetq.core.buffers.impl.ChannelBufferWrapper.setIndex(ChannelBufferWrapper.java:500)

at org.hornetq.core.message.impl.MessageImpl.encodeToBuffer(MessageImpl.java:941)

at org.hornetq.core.message.impl.MessageImpl.getEncodedBuffer(MessageImpl.java:510)

at org.hornetq.core.protocol.core.impl.wireformat.SessionSendMessage.encode(SessionSendMessage.java:68)

at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:235)

at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:210)

at org.hornetq.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:338)

at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:304)

at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:135)

at org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:718)

at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:600)

at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2604)

at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2065)

at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2017-08-02 17:06:46,100 ERROR [Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ224041: Failed to deliver: java.util.NoSuchElementException

at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:178)

at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2087)

at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2017-08-14 05:53:41,105 WARN [Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ222151: removing consumer which did not handle a message, consumer=ClusterConnectionBridge@58d6b42e [name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, queue=QueueImpl[name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1cf0a93d targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@58d6b42e [name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, queue=QueueImpl[name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1cf0a93d targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****e&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1086091769[nodeUUID=f9723119-e8f9-11e6-a77b-3d7585531af6, connector=TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****e&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true, address=jms, server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6])) [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true], discoveryGroupConfiguration=null]], message=Reference[34442548418]:NON-RELIABLE:ServerMessage[...]@1052176512: java.lang.IndexOutOfBoundsException: readerIndex: 17, writerIndex: 0 (expected: 0 <= readerIndex <= writerIndex <= capacity(0))

at io.netty.buffer.AbstractByteBuf.setIndex(AbstractByteBuf.java:99)

at io.netty.buffer.WrappedByteBuf.setIndex(WrappedByteBuf.java:116)

at org.hornetq.core.buffers.impl.ChannelBufferWrapper.setIndex(ChannelBufferWrapper.java:500)

at org.hornetq.core.message.impl.MessageImpl.<init>(MessageImpl.java:170)

at org.hornetq.core.message.impl.MessageImpl.<init>(MessageImpl.java:134)

at org.hornetq.core.server.impl.ServerMessageImpl.<init>(ServerMessageImpl.java:86)

at org.hornetq.core.server.impl.ServerMessageImpl.copy(ServerMessageImpl.java:212)

at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.beforeForward(ClusterConnectionBridge.java:180)

at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:574)

at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2604)

at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2065)

at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2017-08-14 05:53:41,108 ERROR [Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ224041: Failed to deliver: java.util.NoSuchElementException

at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:178)

at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2087)

at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

I am using WildFly 8.2.1 Final and HornetQ 2.4.5.Final.

Does any one face the same problem? I really appreciate if there is any help!

Thanks,

Nghiem

1. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

jbertram Aug 15, 2017 10:25 AM (in response to bvnghiem1012)

I'm not sure what could cause this kind of error. Do you have a reproducible test-case? Have you tried using a more recent version of Wildfly?
Actions
2. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 15, 2017 10:31 PM (in response to jbertram)

Hi J.Bertram,
Thank for your reply!

This error happens on our old product at customer setup occasionally so we don't have a reproducible test case. This is the first time we see this for 3 years with this product.

I can see this exception was happening at setting buffer indexes. Do you think this is buffer problem?
We have set below configuration to disable direct buffer because it caused another memory problem:
-Dio.netty.recycler.maxCapacity.default=0 -Dio.netty.allocator.numDirectArenas=0 -Dio.netty.tryUnsafe=false

Beside that, netty is also upgraded to 4.0.26.Final other than the original version of WildFly 8.2.1.

We don NOT observe this issue on newer versions of our product with WildFly 9 and WildFly 10.

Thanks,
Nghiem
Actions
3. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 15, 2017 11:18 PM (in response to bvnghiem1012)

There is a similar topic on Netty forum, it turned out to a concurrency problem: Google Groups
Tim Fox says:
"In my experience this is the kind of exception I have seen when Netty buffers are used concurrently by different threads."
Actions
4. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 16, 2017 4:28 AM (in response to jbertram)

Hi J.Bertram. I found this ticket. But it is fixed at 2.4.0: https://issues.jboss.org/browse/HORNETQ-1264
Actions
5. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 16, 2017 4:29 AM (in response to bvnghiem1012)

[HORNETQ-1264] org.jboss.netty.buffer.AbstractChannelBuffer.setIndex(AbstractChannelBuffer.java:63) - JBoss Issue Tracke…
Actions
6. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 16, 2017 5:06 AM (in response to bvnghiem1012)

I also found further commit on this message copy fixing: https://github.com/apache/activemq-artemis/commit/e62a820414bae8e5bfaf52febdb6f0edca19489e
Actions
7. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

bvnghiem1012 Aug 16, 2017 5:40 AM (in response to bvnghiem1012)

Hi j.Bertram,

Can you confirm if the changset in my previous comment will help us solve this issue?
Actions
8. Re: [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

jbertram Aug 16, 2017 11:13 AM (in response to bvnghiem1012)

I can't confirm that any of those issues are related to yours since I don't understand the underlying issue. You're certainly free to try the fixes from those issues out and see if it helps, but if you don't have a reliable reproducer it may be tough to prove.

At this point there's not much that will be done for such an old release. In general I recommend you stay up to date with the latest versions.
1 of 1 people found this helpful
Actions

Go to original post