1 Reply Latest reply on Aug 24, 2010 3:37 PM by justinkwaugh

    Message delivery fails after cluster member re-joins

    justinkwaugh

      I have 3 servers (A,B,C) set up in a chained cluster.  Connections to each other are defined explicitly.

       

      A connects to B.

      B connects to A and C.

      C connects to B.

       

      The cluster connections on all three are defined to cluster the address jms.

       

      Server A cluster config looks like this:

      <cluster-connection name="test-cluster">

        <address>jms</address>    
        <max-hops>2</max-hops>
        <connector-ref connector-name="server-b"/>
      </cluster-connection>
      Server B cluster config looks like this:

      <cluster-connection name="test-cluster">

        <address>jms</address>    
        <max-hops>2</max-hops>
        <connector-ref connector-name="server-a"/>
        <connector-ref connector-name="server-c"/>
      </cluster-connection>
      Server C cluster config looks like this:

      <cluster-connection name="test-cluster">

        <address>jms</address>    
        <max-hops>2</max-hops>
        <connector-ref connector-name="server-b"/>
      </cluster-connection>

       

      I have no queues or topics defined on Server B.  On server A and C I have a jms queue defined:

       

       

      <queue name="testQueue">

          <entry name="/queue/testQueue"/>

      </queue>

       

      I can start up A, B, C and connect a producer and consumer and everything works well.  Then I kill server B, and start it back up.  I get an exception (pasted below) and no more messages are delivered.  If I then stop C and start it again, and reconnect the consumer, messages flow again, but any sent during that down period are not consumed.

       

      [Thread-7 (group:HornetQ-client-global-threads-1937054)] 15:27:52,140 SEVERE [or
      g.hornetq.core.server.cluster.impl.ClusterConnectionImpl]  Failed to handle mess
      age
      java.lang.IllegalStateException: Cannot find binding for jms.queue.testQueue46c46
      fe4-aa18-11df-a3ac-005056940757
              at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlo
      wRecordImpl.doConsumerClosed(ClusterConnectionImpl.java:828)
              at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlo
      wRecordImpl.onMessage(ClusterConnectionImpl.java:574)
              at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientC
      onsumerImpl.java:823)
              at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientCons
      umerImpl.java:46)
              at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientCons
      umerImpl.java:941)
              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(Ordere
      dExecutorFactory.java:96)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
      utor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
      .java:908)
              at java.lang.Thread.run(Thread.java:619)
      [Thread-7 (group:HornetQ-client-global-threads-1937054)] 15:28:21,766 SEVERE [or
      g.hornetq.core.server.cluster.impl.ClusterConnectionImpl]  Failed to handle mess
      age
      java.lang.IllegalStateException: Cannot find binding for jms.queue.testQueue46c46
      fe4-aa18-11df-a3ac-005056940757
              at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlo
      wRecordImpl.doConsumerCreated(ClusterConnectionImpl.java:774)
              at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$MessageFlo
      wRecordImpl.onMessage(ClusterConnectionImpl.java:568)
              at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientC
      onsumerImpl.java:823)
              at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientCons
      umerImpl.java:46)
              at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientCons
      umerImpl.java:941)
              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(Ordere
      dExecutorFactory.java:96)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
      utor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
      .java:908)
              at java.lang.Thread.run(Thread.java:619)

       

      I also see this message every five seconds on server B with the command id incrementing :

       

      [Old I/O client worker (channelId: 20259911, /(server b's ip):3863 => (server c's host name/server c's ip:5445)] 15:32:44,255 WARNING [org.hornetq.core.protocol

      .core.impl.ChannelImpl]  Can't find packet to clear:  last received command id 5

      8 first stored command id 0