6 Replies Latest reply on Nov 14, 2012 3:08 AM by Andy Taylor

    JMX for clustered HornetQ

    tim.rombauts Newbie

      I'm running HornetQ in a cluster, on 2 separate machines.

      The cluster is made using static connectors.

       

      This works fine: both servers are processing JMS messages from the queues, and distribution seems almost 50-50.

       

      However, I'd like to use the JMX exposed data to monitor the cluster's health and status.

      At first startup of the cluster, these JMX counters are fine; messageCount, deliveringCount and consumerCount seems ok.

       

      However, from the moment I restart one of the servers, the counters in the restarted server remain 0.

      This restarted server is however processing messages from the queue just fine, although JMX doesn't indicate that.

       

      Anything I'm doing wrong here?

       

      Thanks

        • 1. Re: JMX for clustered HornetQ
          Andy Taylor Master

          that implies to me that your messages aren't persisted so there aren't any messages reloaded from the journal on restart, can you confirm if this is the case.

          • 2. Re: JMX for clustered HornetQ
            tim.rombauts Newbie

            The hornetq folder does contain journal files (each 10MB, indexes -1, -2, -3, -4, -5 and -10) which have all been touched recently.

            So I guess persistency of the journal is working ok.

             

            1 remark: server is using NIO, not AIO.

            • 3. Re: JMX for clustered HornetQ
              Clebert Suconic Master

              What Andy said was.. are you sure you sent your messages as persisted, and you created the queues as persisted?

               

               

              Take a look on the PrintData and PrintPages (if you're using Pages at all).

               

               

              #java -cp /somewhere/hornetq-logging.jar:/somewhere/hornetq-core.jar:/somewhere/netty.jar org.hornetq.core.persistence.impl.journal.PrintData $1 $2

               

               

               

              BTW: this is not on the docs because I've created as a debug tool for myself and it's becoming used by other people as well. I will make sure I add it to the docs for next version.

              • 4. Re: JMX for clustered HornetQ
                tim.rombauts Newbie

                I now made the queue durable and set deliveryPersistent to true.

                 

                The behavior remains the same: on 1 of the servers, the consumerCount and messagesAdded remain 0, although this server is processing 50% of the messages of the clustered queue.

                 

                I also did a second test: once I stop the server where the JMX counters were ok, the JMX counters of the erroneous server suddenly start to show up fine.

                Restart the other server again, makes the JMX counters in that server remain 0.

                 

                PrintData of the journal does seem to indicate messages are be durable, numberOfRecords also seems to indicate this, see below.

                 

                Anything else I can look into?

                 

                Thanks!!!

                 

                 

                ===> Server 1

                 

                operation@AddRecordTX;txID=2155737966,recordID=2155737967;userRecordType=31;isUpdate=false;Message(messageID=2155737967;properties=[#properties = 0] - ServerMessage[messageID=2155737967,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@521594489

                operation@UpdateTX;txID=2155737966,recordID=2155737967;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

                operation@Commit;txID=2155737966,numberOfRecords=2

                operation@UpdateTX;txID=2155737945,recordID=2155737967;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

                operation@Commit;txID=2155737945,numberOfRecords=1

                operation@DeleteRecord;recordID=2155737967

                operation@AddRecordTX;txID=2155738060,recordID=2155738061;userRecordType=31;isUpdate=false;Message(messageID=2155738061;properties=[#properties = 0] - ServerMessage[messageID=2155738061,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@410329569

                operation@UpdateTX;txID=2155738060,recordID=2155738061;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

                operation@Commit;txID=2155738060,numberOfRecords=2

                operation@UpdateTX;txID=2155738053,recordID=2155738061;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

                operation@Commit;txID=2155738053,numberOfRecords=1

                 

                 

                ===> Server 2

                 

                operation@AddRecordTX;txID=2157572882,recordID=2157572883;userRecordType=31;isUpdate=false;Message(messageID=2157572883;properties=[#properties = 0] - ServerMessage[messageID=2157572883,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@179331370

                operation@UpdateTX;txID=2157572882,recordID=2157572883;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

                operation@Commit;txID=2157572882,numberOfRecords=2

                operation@UpdateTX;txID=2157572876,recordID=2157572883;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

                operation@Commit;txID=2157572876,numberOfRecords=1

                operation@DeleteRecord;recordID=2157572883

                operation@AddRecordTX;txID=2157572932,recordID=2157572933;userRecordType=31;isUpdate=false;Message(messageID=2157572933;properties=[#properties = 0] - ServerMessage[messageID=2157572933,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@1102778849

                operation@UpdateTX;txID=2157572932,recordID=2157572933;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

                operation@Commit;txID=2157572932,numberOfRecords=2

                operation@UpdateTX;txID=2157572929,recordID=2157572933;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

                operation@Commit;txID=2157572929,numberOfRecords=1

                operation@DeleteRecord;recordID=2157572933

                • 5. Re: JMX for clustered HornetQ
                  tim.rombauts Newbie

                  Does anyone have an idea where to look next?

                  • 6. Re: JMX for clustered HornetQ
                    Andy Taylor Master

                    I now made the queue durable and set deliveryPersistent to true.

                    How did you set delivery to persistant?

                    The behavior remains the same: on 1 of the servers, the consumerCount and messagesAdded remain 0, although this server is processing 50% of the messages of the clustered queue.

                    if messages added is 0 then this server has processes none of the messages

                     

                    PrintData of the journal does seem to indicate messages are be durable, numberOfRecords also seems to indicate this, see below

                    The print data also shows the messages acked and the records deleted, i.e. consumed.

                     

                    Why don't you provide a simple test that we can look at, with easy to follow instructions.

                     

                    Also make sure you are using the latest version