5 Replies Latest reply on Jul 24, 2009 8:17 AM by ataylor

    Queue client failover solution while jboss5.1 crashed

      As nobody answer my question on another post with different meaning subject,I created a new topic here.

      I would like to implement this solution via JBM2 that we have a cluster with multiple nodes been connected each other,we deploy queue on every node with same queue name to form a messaging cluster,we deploy queue producer on several nodes and also deploy consumer on some nodes as well with messaging load balance(server side),our requirement is once one node down(Jboss5.1 completely crashed ) which has consumer ,the other consumers will consume the messaging sent to down server,once this server up ,it will reattched into the messaging cluster and resume to consume the queue messages as messages been load balanced to this node again.

      I have configured the messaging cluster and queue messaging redistribution,but seems it didn't work as I expected.

      For example we have 2 nodes configured to be a messaging cluster,server1 has one producer and one consumer,and server2 has one consumer.

      Sometime,after I shutdown server2,and sent 10 messages to the queue,server1 only consume 5 of them,looks like redistribution failed.Once I restarted the server2,it got rest of 5 messages!

      I also keep getting this WARN message after I shutdown/restart 2nd server 3 times,
      WARNING [DiscoveryGroupImpl] There seem to be more than one broadcasters on the network broadcasting the same node id

      Any input will be highly appreciated.

        • 1. Re: Queue client failover solution while jboss5.1 crashed
          ataylor

           

          As nobody answer my question on another post with different meaning subject,I created a new topic here.


          shouting on lots of topics won't get you an answer any quicker. The development team are very busy but will get round to answering them in time. see below!

          You are confusing redistribution with failover. Redistribution will automatically re route its messages when a queue has no consumer. In your case you are killing the node so obviously it cant redistribute. When it comes back up the messages are there so it just sends to the consumer when it reconnects.

          What you want is failover. basically this requires a back up node being configured. Have a look at the automatic-failover and symmetric-cluster examples to see how this is done. Also read the section in the user manual.

          I also keep getting this WARN message after I shutdown/restart 2nd server 3 times,



          • 2. Re: Queue client failover solution while jboss5.1 crashed

            Taylor,thanks very much for your prompt response!

            I understood your point,but could you pls tell me how I can get rid of that WARN messging,

            WARNING [DiscoveryGroupImpl] There seem to be more than one broadcasters on the network broadcasting the same node id

            It looks like once I started Jboss and JBM got redeployed in the cluster,but it used the same broadcaster name as it should be while I hard configured in the jbm-configuration file,but old broadcaster group didn't get removed from system somehow when I killed the node.

            BTW I have setup <connection-ttl-override>10000</connection-ttl-override> ,so once one node down,server will clean up resources like connection for that node after 10 sec,but how can JBM remove broadcaster group on that node?

            • 3. Re: Queue client failover solution while jboss5.1 crashed
              ataylor

              The timeout is configured in the discovery group, i.e.

              <refresh-timeout>10000</refresh-timeout>


              default is 10 second s I think. make sure you wait until 2 * this setting before restarting the node.



              • 4. Re: Queue client failover solution while jboss5.1 crashed

                I have set <refresh-timeout>5000</refresh-timeout> for 5 secs ,and wait for at least 15 secs to start node but still throw up same WARN message,I also got below exceptions

                02:49:59,627 SEVERE [ServerSessionImpl] Failed to send message
                java.lang.NullPointerException
                at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
                at org.jboss.messaging.core.postoffice.impl.BindingsImpl.routeFromCluster(BindingsImpl.java:139)
                at org.jboss.messaging.core.postoffice.impl.BindingsImpl.route(BindingsImpl.java:284)
                at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:656)
                at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:667)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:2526)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2334)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleSend(ServerSessionImpl.java:1038)
                at org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:269)
                at org.jboss.messaging.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:621)
                at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:399)
                at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:483)
                at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:75)
                at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
                at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:387)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.fireMessageReceived(FrameDecoder.java:284)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:261)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:182)
                at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:344)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:331)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:303)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:255)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:176)
                at org.jboss.netty.util.internal.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
                at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:49)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)

                Sometimes once I restarted the server been killed,I got following exceptions,

                03:06:51,359 WARNING [ClientSessionImpl] 1670189749 Session not found on server when attempting to re-attach
                03:06:51,370 SEVERE [BridgeImpl] Failed to stop
                java.lang.IllegalStateException: Cannot find add info 121
                at org.jboss.messaging.core.journal.impl.JournalImpl.appendUpdateRecord(JournalImpl.java:320)
                at org.jboss.messaging.core.persistence.impl.journal.JournalStorageManager.updateDeliveryCount(JournalStorageManager.java:462)
                at org.jboss.messaging.core.server.impl.QueueImpl.checkDLQ(QueueImpl.java:1077)
                at org.jboss.messaging.core.server.impl.QueueImpl.cancel(QueueImpl.java:688)
                at org.jboss.messaging.core.server.cluster.impl.BridgeImpl.cancelRefs(BridgeImpl.java:299)
                at org.jboss.messaging.core.server.cluster.impl.BridgeImpl.access$800(BridgeImpl.java:79)
                at org.jboss.messaging.core.server.cluster.impl.BridgeImpl$FailRunnable.run(BridgeImpl.java:756)
                at org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:105)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
                03:06:51,671 SEVERE [ServerSessionImpl] Failed to send message
                java.lang.NullPointerException
                at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
                at org.jboss.messaging.core.postoffice.impl.BindingsImpl.routeFromCluster(BindingsImpl.java:139)
                at org.jboss.messaging.core.postoffice.impl.BindingsImpl.route(BindingsImpl.java:284)
                at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:656)
                at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:667)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:2526)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2334)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleSend(ServerSessionImpl.java:1038)
                at org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:269)
                at org.jboss.messaging.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:621)
                at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:399)
                at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:483)
                at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:75)
                at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
                at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:387)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.fireMessageReceived(FrameDecoder.java:284)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:261)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:182)
                at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:344)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:331)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:303)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:255)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:176)
                at org.jboss.netty.util.internal.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
                at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:49)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
                03:06:51,672 SEVERE [ServerSessionImpl] Failed to send message
                java.lang.NullPointerException
                at java.nio.ByteBuffer.wrap(ByteBuffer.java:373)
                at org.jboss.messaging.core.postoffice.impl.BindingsImpl.routeFromCluster(BindingsImpl.java:139)
                at org.jboss.messaging.core.postoffice.impl.BindingsImpl.route(BindingsImpl.java:284)
                at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:656)
                at org.jboss.messaging.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:667)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:2526)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2334)
                at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleSend(ServerSessionImpl.java:1038)
                at org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:269)
                at org.jboss.messaging.core.remoting.impl.ChannelImpl.handlePacket(ChannelImpl.java:621)
                at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:399)
                at org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:483)
                at org.jboss.messaging.integration.transports.netty.MessagingChannelHandler.messageReceived(MessagingChannelHandler.java:75)
                at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:105)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
                at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:803)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:387)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.fireMessageReceived(FrameDecoder.java:284)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:261)
                at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:182)
                at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:87)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:567)
                at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:562)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:344)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:331)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:303)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:255)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:176)
                at org.jboss.netty.util.internal.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:72)
                at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:49)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)

                • 5. Re: Queue client failover solution while jboss5.1 crashed
                  ataylor

                  If you could provide a standalone test showing this that would be great. Attach it to a JIRA and someone will take a look.