JBossDeveloper

Justin Bertram wrote:

 

That test is not the one I am comparing anyway as the results were so bad that it was clear the while/true loop was resulting in an unfair test.

I'm a little confused.  It seems like you're saying that the code you provided here is not actually the code you're using in your benchmark.  Is that true?  If so, why did you provide it here?

 

Sorry for the confusion. The test runs in a few modes. The while/true loop with HornetQ being one of those (as the receiver program was exiting unless I kept the main thread alive), but it just uses CPU, so I'd have to put a sleep in the loop for a duration longer than the sending program runs. I did make that change after making the above post, but I decided to run the test in the mode that uses MessageConsumer.receive() rather than in the mode that uses a MessageListener and a while/true loop.

We use MySQL in production with JBossMQ. I can make that change to the JBossMQ test scenario, as well as the two settings, if it will make a fairer comparison?

I do think using MySQL in your benchmarking of JBossMQ would be more fair if you can ensure that MySQL is syncing to disk on every insert/delete.  As it stands now you're using Hypersonic which doesn't sync to disk on every insert/delete.  In fact, the version of Hypersonic included in JBoss AS 4.0.4 is more akin to a toy database (no offense to the authors of HSQLDB).

 

Yes, I agree, I should have used MySQL for my test. I will do that.

 

Quick question...In your production environment is MySQL physically located on the same box as JBoss AS?

 

Yes. There is federation to databases on other boxes, as well as remote databases involved, but if you are talking about the jms_messages etc tables, then yes, they are local on each machine.

Does HornetQ have a database behind it (like the jms_messages etc table in JBossMQ)?

No.  HornetQ uses a local journal file rather than a database.

Turns out that the JBossMQ tests I have been doing are using MySQL, and not HSQLDB. I had taken a jboss 4.0.4 installation from one of our production machines and cleared everything out of it, but the use of MySQL with JBossMQ remained.

So all the original numbers were a fair test it seems? And Clebert's comment that "JBoss MQ with HSQL doesn't block on syncs" is not relevant. A record is written to the jms_messages table (which is InnoDB type) in the MySQL database for every message sent (I had assumed I could control that behavior with MessageCache's HighMemoryMark and MaxMemoryMark, but chaning the value of these does not make a difference to what is written to jms_messages). I guess that means that this condition that Clebert mentioned for a fair test is also met "...make it perform syncs on committs" (I can find no mention of a specific 'sync on commit' setting)?

I would still try the two connection-factory settings Clebert mentioned, but Googling for what the configuration parameters are called yielded block-on-persistent-send and block-on-acknowledge. The 2.2.5 documentation here http://docs.jboss.org/hornetq/2.2.5.Final/user-manual/en/html/configuration-index.htmlmentions only the latter setting, and it has a default of false anyway! I can see mention of block-on-persistent-send in documentation for older versions.

I just came back to this. Setting block-on-durable-send to false as Clebert suggested does indeed significantly increase the throughput of HornetQ. And I agree that it must be the case that JBossMQ is not honoring store-and-forward (edit - actually, as I already mentioned above, JBossMQ IS writing every message to the jms_messages MySQL Innodb table for every message - I can be sure of that because I created an on-insert trigger on that table).

block-on-durable-send is set in hornetq-jms.xml. I was not sure which of the three Connection Factories defined in there (NettyConnectionFactory, NettyThroughputConnectionFactory, InVMConnectionFactory) was the one I should change, so I printed out the class returned by (ConnectionFactory)context.lookup("ConnectionFactory"), and it printed:

          org.hornetq.jms.client.HornetQJMSConnectionFactory

so that didn't help identify which one. A bit more debug showed that NettyConnectionFactory was the section I needed to change.

The documentation states: "HornetQ can be configured to block the call to send until the message has definitely reached the server, and a response has been sent back to the client".

Given that block-on-durable-send is set against a Connection Factory, then is the block enforced at the Connection level? So if I have two threads, each with their own Connection, then Thread 2 will not be prevented from sending just because Thread 1's send has not been acknowledged yet? Or is the block enforced at the server side?

The reason I ask is that I am trying to understand the difference I see in the two runs between block-on-durable-send=true/false. The sender is multi-threaded; the sender threads synchronize to get a sequential number for the content of the TextMessage they send. The receiver prints that number. With block-on-durable-send=true, after the first 100 or so messages that the receiver consumes, all subsequent messages are in numerical order. That is not the case when block-on-durable-send=false, where the messages are significanlty out of order (by thousands). I had expected the run with block-on-durable-send=true to have been out-of-order too, but perhaps if the blocking is done at the server, and context switching is frequent enough, then the ordering is not unexpected?

This thread https://community.jboss.org/thread/164769 introduced me to two new performance-related settings:

          journal-sync-transactional

          journal-sync-non-transactional

I haven't investigated their meaning yet, and I'm not clear on how they interact, if at all, with block-on-[non]durable-send. Setting the above two to false increases the number of messages I can push through HornetQ, as does block-on-durable-send=false, but setting all three to false reduces the total number back down. Maybe I just need to run more tests, but currently getting:

block-on-durable-send=true                              3500

block-on-durable-send=false                              10500

journal-* = false                                        20000

journal-* = false + block-on-durable-send=false          7500

The descriptions of block-on-durable-send (which I think would be more correctly named block-on-persistent-send) and block-on-non-durable-send are almost identical. The documentation for block-on-durable-send is:

"If this is set to true then all calls to send for durable messages on non transacted sessions will block until the message has reached the server, and a response has been sent back."

I wondered why the description did not say:

"If this is set to true then all calls to send for durable messages on non transacted sessions will block until the message has reached the server, persisted to the journal, and a response has been sent back."

I assumed it was just down to the documentation not being very explicit, but then I looked at the description of journal-sync-non-transactional:

"If the journal parameter journal-sync-non-transactional is set to true the server will not send a response back to the client until the message has been persisted and the server has a guarantee that the data has been persisted to disk"

If the journal-sync-non-transactional setting is controlling the persistence aspect of the blocking, what's the point in having two parameters block-on[-non]-durable-send?

And I have just noticed that a Queue can be configured as durable or not - so even if you send with delivery mode = persistent, that will be ignored if the queue is not configured as durable?

Hello,

I have done a few tests try to measure the effect of the block-on-durable-send and journal-sync-non-transactional settings. I think that block-on-acknowledge is relevant, but I'm not sure what it means yet, so I haven't looked at that one.

In all tests, Delivery Mode = PERSISTENT, Session transacted = false, Session ack = AUTO ACK, Queue = durable.

My understandig from reading the documentation is that since block-on-durable-send is set against a connection factory, the connection handed to a client application will carry that fact. And as journal-sync-non-transaction is set in hornetq-configuration.xml, so the client application probably has no visibility of that setting? With block-on-durable-send = true, when a message is sent the client is blocked from sending another message until the server has acknowledged. If journal-sync-non-transactional is set to false, that acknowledgement will be sent as soon as the server gets the message. If journal-sync-non-transactional is set to true, the server will ensure that the message is persisted to storage before sending the acknowledgement.

But one question I was interested in was the effect, if any, of journal-sync-non-transactional when block-on-durable-send = false. I have a single threaded Java program taking messages off the queue, and I wondered if the rate that it takes messages off the queue would be slower when journal-sync-non-transactional = true because the server would have to do some extra work of some sort before the message was available to the receiver? Or whether journal-sync-non-transactional solely affects the point at which an ack is sent to the client, and therefore has no affect on the receiver (and in fact has no effect on the sender either when block-on-durable-send is set to false)?

I was almost certain that journal-sync-non-transactional would have no affect on the message rate I was able to push through when block-on-durable-send = false, but it does seem to have an effect - being faster when journal-sync-non-transactional = false. I know my benchmarking code would not stand up to any examination, but I think it is kinda good enough. Given what I said above, although I'd be surprised, I suppose I could understand if my receiver took longer to consume the messages on the queue. But the number of messages my sender program (running on the same machine) sends should be unaffected? And yet it sends more when journal-sync-non-transactional = false. Sorry, I know I'm missing something. Could someone tell me if I should expect the behaviour I am seeing?

I'm hoping there is an explanation without having to go into detail of how my sender/receiver work, but I'd better give a brief description of them - the sender has 5 threads doing the sending, each with their own connection/seesion. Each thread loops for 30 seconds, sending text messages. The receiver is single-threaded and loops forever doing a MessageConsumer.receive(0). It measures the time it took to process the messages it received by taking the time difference between the first message it received and the last message it received. I crudely take into account the fact that the sending threads may have started sending messages at different times when calculating the time taken to receive messages. The results I got from 5 runs are:

block-on-durable-send = false

journal-sync-non-transactional = true

1. Sender                    188102
   Receiver          188102 in 27.795 seconds
2. Sender                    228135
   Receiver          228135 in 30.457 seconds
3. Sender                    184957
   Receiver          184957 in 30.755 seconds
4. Sender                    192650
   Receiver          192650 in 29.359 seconds
5. Sender                    210016
   Receiver          210016 in 30.515 seconds

block-on-durable-send = false

journal-sync-non-transactional = false :

1. Sender                    277801
   Receiver          277801 in 30.740 seconds
2. Sender                    281709
   Receiver          281709 in 31.095 seconds
3. Sender                    285662
   Receiver          285662 in 31.368 seconds
4. Sender                    291358
   Receiver          291358 in 34.006 seconds
5. Sender                    288074
   Receiver          288074 in 29.976 seconds

There is one other odd result that I can't explain. Sometimes the messages seem to trickle in to the receiver. For example, on one run the sender reported that it had sent 169794 messages. I killed my receiver and it reported it had received 163262 message in 30 seconds (the receiver uses a finalizer to report number of messages it received in that run). So I restarted the receiver to get the rest of the messages, and the messages seemed to trickle in:

          0 in 0 seconds                    (receiver running for approx 10s)

          1633 in 1 seconds          (receiver running for approx 30s)

          2449 in 1 seconds          (receiver running for approx 30s)

          1225 in 0 seconds          (receiver running for approx 30s)

          613 in 0 seconds          (receiver running for approx 30s)

          306 in 0 seconds          (receiver running for approx 30s)

          0 in 0 seconds                    (receiver running for approx 30s)

          0 in 0 seconds                    (receiver running for approx 30s)

          153 in 0 seconds          (receiver running for approx 60s)

          153 in 0 seconds          (receiver running for approx 60s)

...and now the receiver has received all the 169794 messages the sender sent. I've seen this behavior on more than one occasion. Anyone got an explanation? I'm not going to get too worried about it, but I just thought if someone could offer a viable explanation it may help me understand the workings of HornetQ a bit better.

Thank you for any help,

Paul