8 Replies Latest reply on Aug 16, 2017 11:13 AM by Justin Bertram

    [WildFly 8.2.1.Final] Cluster Connection Bridge consumer is removed. Messages are not forwarded between cluster nodes

    Bui Van Nghiem Newbie

      Hi experts,

       

      I am facing the problem that messages are not forwarded between 2 cluster nodes. I check the server.log and below exceptions are seen on 1 server. It happen 2 times for 2 different cluster nodes which are connected to current server:

       

      2017-08-02 17:06:46,096 WARN  [Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ222151: removing consumer which did not handle a message, consumer=ClusterConnectionBridge@fff7f6e [name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, queue=QueueImpl[name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1ce980b3 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@fff7f6e [name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, queue=QueueImpl[name=sf.cluster-connection.f4524855-e8fa-11e6-b619-9164db24d580, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1ce980b3 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?key-store-password=****&batch-delay=200&port=5455&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****e&tcp-receive-buffer-size=200000&nio-remoting-threads=5&use-nio=true], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1086091769[nodeUUID=f9723119-e8f9-11e6-a77b-3d7585531af6, connector=TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true, address=jms, server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6])) [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?key-store-password=****&batch-delay=200&port=5455&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****&tcp-receive-buffer-size=200000&nio-remoting-threads=5&use-nio=true], discoveryGroupConfiguration=null]], message=Reference[34376129663]:NON-RELIABLE:ServerMessage[...]@498949710: java.lang.IndexOutOfBoundsException: readerIndex: 0, writerIndex: 5348 (expected: 0 <= readerIndex <= writerIndex <= capacity(17))

          at io.netty.buffer.AbstractByteBuf.setIndex(AbstractByteBuf.java:99)

          at io.netty.buffer.WrappedByteBuf.setIndex(WrappedByteBuf.java:116)

          at org.hornetq.core.buffers.impl.ChannelBufferWrapper.setIndex(ChannelBufferWrapper.java:500)

          at org.hornetq.core.message.impl.MessageImpl.encodeToBuffer(MessageImpl.java:941)

          at org.hornetq.core.message.impl.MessageImpl.getEncodedBuffer(MessageImpl.java:510)

          at org.hornetq.core.protocol.core.impl.wireformat.SessionSendMessage.encode(SessionSendMessage.java:68)

          at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:235)

          at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:210)

          at org.hornetq.core.client.impl.ClientProducerImpl.sendRegularMessage(ClientProducerImpl.java:338)

          at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:304)

          at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:135)

          at org.hornetq.core.server.cluster.impl.BridgeImpl.deliverStandardMessage(BridgeImpl.java:718)

          at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:600)

          at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2604)

          at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2065)

          at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

          at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

       

      2017-08-02 17:06:46,100 ERROR [Thread-24 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ224041: Failed to deliver: java.util.NoSuchElementException

          at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:178)

          at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2087)

          at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

          at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

       

      2017-08-14 05:53:41,105 WARN  [Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ222151: removing consumer which did not handle a message, consumer=ClusterConnectionBridge@58d6b42e [name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, queue=QueueImpl[name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1cf0a93d targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@58d6b42e [name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, queue=QueueImpl[name=sf.cluster-connection.04098dd6-e902-11e6-bce1-9de8090f0177, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6]]@1cf0a93d targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****e&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1086091769[nodeUUID=f9723119-e8f9-11e6-a77b-3d7585531af6, connector=TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****e&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true, address=jms, server=HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6])) [initialConnectors=[TransportConfiguration(name=netty-throughput, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5455&batch-delay=200&key-store-password=****&host=****&use-nio-global-worker-pool=true&tcp-send-buffer-size=200000&ssl-enabled=true&key-store-path=****&nio-remoting-threads=5&tcp-receive-buffer-size=200000&use-nio=true], discoveryGroupConfiguration=null]], message=Reference[34442548418]:NON-RELIABLE:ServerMessage[...]@1052176512: java.lang.IndexOutOfBoundsException: readerIndex: 17, writerIndex: 0 (expected: 0 <= readerIndex <= writerIndex <= capacity(0))

          at io.netty.buffer.AbstractByteBuf.setIndex(AbstractByteBuf.java:99)

          at io.netty.buffer.WrappedByteBuf.setIndex(WrappedByteBuf.java:116)

          at org.hornetq.core.buffers.impl.ChannelBufferWrapper.setIndex(ChannelBufferWrapper.java:500)

          at org.hornetq.core.message.impl.MessageImpl.<init>(MessageImpl.java:170)

          at org.hornetq.core.message.impl.MessageImpl.<init>(MessageImpl.java:134)

          at org.hornetq.core.server.impl.ServerMessageImpl.<init>(ServerMessageImpl.java:86)

          at org.hornetq.core.server.impl.ServerMessageImpl.copy(ServerMessageImpl.java:212)

          at org.hornetq.core.server.cluster.impl.ClusterConnectionBridge.beforeForward(ClusterConnectionBridge.java:180)

          at org.hornetq.core.server.cluster.impl.BridgeImpl.handle(BridgeImpl.java:574)

          at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2604)

          at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2065)

          at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

          at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

       

      2017-08-14 05:53:41,108 ERROR [Thread-20 (HornetQ-server-HornetQServerImpl::serverUUID=f9723119-e8f9-11e6-a77b-3d7585531af6-204988475)]-[org.hornetq.core.server] HQ224041: Failed to deliver: java.util.NoSuchElementException

          at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:178)

          at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:2087)

          at org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:81)

          at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2917)

          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

         

      I am using WildFly 8.2.1 Final and HornetQ 2.4.5.Final.

      Does any one face the same problem? I really appreciate if there is any help!

       

      Thanks,

      Nghiem