JMX for clustered HornetQ

[tim.rombauts]

I'm running HornetQ in a cluster, on 2 separate machines.

The cluster is made using static connectors.

This works fine: both servers are processing JMS messages from the queues, and distribution seems almost 50-50.

However, I'd like to use the JMX exposed data to monitor the cluster's health and status.

At first startup of the cluster, these JMX counters are fine; messageCount, deliveringCount and consumerCount seems ok.

However, from the moment I restart one of the servers, the counters in the restarted server remain 0.

This restarted server is however processing messages from the queue just fine, although JMX doesn't indicate that.

Anything I'm doing wrong here?

Thanks

[ataylor]

that implies to me that your messages aren't persisted so there aren't any messages reloaded from the journal on restart, can you confirm if this is the case.

[tim.rombauts]

The hornetq folder does contain journal files (each 10MB, indexes -1, -2, -3, -4, -5 and -10) which have all been touched recently.

So I guess persistency of the journal is working ok.

1 remark: server is using NIO, not AIO.

[clebert.suconic]

What Andy said was.. are you sure you sent your messages as persisted, and you created the queues as persisted?

Take a look on the PrintData and PrintPages (if you're using Pages at all).

#java -cp /somewhere/hornetq-logging.jar:/somewhere/hornetq-core.jar:/somewhere/netty.jar org.hornetq.core.persistence.impl.journal.PrintData $1 $2

BTW: this is not on the docs because I've created as a debug tool for myself and it's becoming used by other people as well. I will make sure I add it to the docs for next version.

[tim.rombauts]

I now made the queue durable and set deliveryPersistent to true.

The behavior remains the same: on 1 of the servers, the consumerCount and messagesAdded remain 0, although this server is processing 50% of the messages of the clustered queue.

I also did a second test: once I stop the server where the JMX counters were ok, the JMX counters of the erroneous server suddenly start to show up fine.

Restart the other server again, makes the JMX counters in that server remain 0.

PrintData of the journal does seem to indicate messages are be durable, numberOfRecords also seems to indicate this, see below.

Anything else I can look into?

Thanks!!!

===> Server 1

operation@AddRecordTX;txID=2155737966,recordID=2155737967;userRecordType=31;isUpdate=false;Message(messageID=2155737967;properties=[#properties = 0] - ServerMessage[messageID=2155737967,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@521594489

operation@UpdateTX;txID=2155737966,recordID=2155737967;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

operation@Commit;txID=2155737966,numberOfRecords=2

operation@UpdateTX;txID=2155737945,recordID=2155737967;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

operation@Commit;txID=2155737945,numberOfRecords=1

operation@DeleteRecord;recordID=2155737967

operation@AddRecordTX;txID=2155738060,recordID=2155738061;userRecordType=31;isUpdate=false;Message(messageID=2155738061;properties=[#properties = 0] - ServerMessage[messageID=2155738061,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@410329569

operation@UpdateTX;txID=2155738060,recordID=2155738061;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

operation@Commit;txID=2155738060,numberOfRecords=2

operation@UpdateTX;txID=2155738053,recordID=2155738061;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

operation@Commit;txID=2155738053,numberOfRecords=1

===> Server 2

operation@AddRecordTX;txID=2157572882,recordID=2157572883;userRecordType=31;isUpdate=false;Message(messageID=2157572883;properties=[#properties = 0] - ServerMessage[messageID=2157572883,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@179331370

operation@UpdateTX;txID=2157572882,recordID=2157572883;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

operation@Commit;txID=2157572882,numberOfRecords=2

operation@UpdateTX;txID=2157572876,recordID=2157572883;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

operation@Commit;txID=2157572876,numberOfRecords=1

operation@DeleteRecord;recordID=2157572883

operation@AddRecordTX;txID=2157572932,recordID=2157572933;userRecordType=31;isUpdate=false;Message(messageID=2157572933;properties=[#properties = 0] - ServerMessage[messageID=2157572933,priority=4, bodySize=1600,expiration=0, durable=true, address=jms.queue.jobQueue,properties=TypedProperties[null]]@1102778849

operation@UpdateTX;txID=2157572932,recordID=2157572933;userRecordType=32;isUpdate=true;AddRef;QueueEncoding [queueID=4]

operation@Commit;txID=2157572932,numberOfRecords=2

operation@UpdateTX;txID=2157572929,recordID=2157572933;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=4]

operation@Commit;txID=2157572929,numberOfRecords=1

operation@DeleteRecord;recordID=2157572933

[tim.rombauts]

Does anyone have an idea where to look next?

[ataylor]

I now made the queue durable and set deliveryPersistent to true.

How did you set delivery to persistant?

The behavior remains the same: on 1 of the servers, the consumerCount and messagesAdded remain 0, although this server is processing 50% of the messages of the clustered queue.

if messages added is 0 then this server has processes none of the messages

PrintData of the journal does seem to indicate messages are be durable, numberOfRecords also seems to indicate this, see below

The print data also shows the messages acked and the records deleted, i.e. consumed.

Why don't you provide a simple test that we can look at, with easy to follow instructions.

Also make sure you are using the latest version