-
1. Re: ambiguous and unclear definition in Hornetq user manual
ataylor Apr 8, 2014 9:42 AM (in response to vasco)moved to correct forum
-
2. Re: ambiguous and unclear definition in Hornetq user manual
jbertram Apr 8, 2014 10:20 AM (in response to vasco)The first statement is that by default packets are received are handled by the remoting thread but then later it states that this behavior can be changed by setting async-connection-execution to true, which is set to true by default. Apart from this contradiction...
I will update the documentation to be more clear.
...the statement about the short running operations confuses me even more i.e. how do we know upfront if something is either short running or not?
Whether or not the operation is "short running" is determined by the type of packet it is. These packets will be executed asynchronously if async-connection-execution is true:
- RollbackMessage
- SessionCloseMessage
- SessionCommitMessage
- SessionXACommitMessage
- SessionXAPrepareMessage
- SessionXARollbackMessage
Every other packet will be executed on the remoting thread regardless of the setting for async-connection-execution.
...in my load testing I can not get the broker to scale out vertically. I think this primarily because all the message filtering is done by only on thread and I'm looking for a way to utilize more of the available CPU power to do the filtering.
Couple of points/questions...
- Your bottleneck may not be CPU. Oftentimes the bottleneck is the disk. Do you have specific evidence that indicates CPU is the bottleneck? If so, what is it?
- From what I understand the filtering should be done by more than one thread. Do you have specific evidence that indicates only one thread is being used for filtering? If so, what is it?
- What exactly are you doing in your load test?
-
3. Re: ambiguous and unclear definition in Hornetq user manual
vasco Apr 8, 2014 10:52 AM (in response to jbertram)Whether or not the operation is "short running" is determined by the type of packet it is. These packets will be executed asynchronously if async-connection-execution is true:
- RollbackMessage
- SessionCloseMessage
- SessionCommitMessage
- SessionXACommitMessage
- SessionXAPrepareMessage
- SessionXARollbackMessage
What about "normal" HornetQBytesMessage messages do they fall under one of these categories?
Your bottleneck may not be CPU. Oftentimes the bottleneck is the disk. Do you have specific evidence that indicates CPU is the bottleneck? If so, what is it?
I don't think disk could be a bottle neck in our case since we are not using persistent messaging so I do no expect anything is written to disc.
From what I understand the filtering should be done by more than one thread. Do you have specific evidence that indicates only one thread is being used for filtering? If so, what is it?
I have profiled our running load test using a profiler (yourkit) and what I see is that the remoting thread is constantly bussy filtering and other threads in the pool are sporadically active. I also introduced a breakpoint that prints the thread name every time the filter was evaluated and this was constantly the remoting-thread.
What exactly are you doing in your load test?
We are sending messages to a broker and measuring the time it takes for it to arrive at a client that subscribes using a message selector.
-
4. Re: ambiguous and unclear definition in Hornetq user manual
jbertram Apr 8, 2014 11:34 AM (in response to vasco)What about "normal" HornetQBytesMessage messages do they fall under one of these categories?
You're confusing the JMS implementation with internal HornetQ network packets. All the "messages" I enumerated previously are special "packets" which go over the wire (i.e. over the network) between the client and server to accomplish specific tasks (e.g. close a session, commit a session, rollback an XA transaction, etc.). These messages are categorically different from anything which implements javax.jms.Message.
We are sending messages to a broker and measuring the time it takes for it to arrive at a client that subscribes using a message selector.
- How many producers do you have?
- How many consumers do you have?
- At what rate are you sending the messages?
- Have you investigated direct-deliver?
-
5. Re: ambiguous and unclear definition in Hornetq user manual
vasco Apr 10, 2014 7:04 AM (in response to jbertram)How many producers do you have?
36
How many consumers do you have?
36 creating a total 1300 subscriptions.
At what rate are you sending the messages?
750 msg/sec
Have you investigated direct-deliver?
We are using direct delivery.
I don't think this is relevant for the fact the the broker filters message in one thread right?
-
6. Re: ambiguous and unclear definition in Hornetq user manual
jbertram Apr 10, 2014 11:05 AM (in response to vasco)I don't think this is relevant for the fact the the broker filters message in one thread right?
Of course it is relevant. Understanding the overall use-case is critical when investigating issues like this.
We are using direct delivery.
Do you mean that you're using the default value for direct-deliver (i.e. true)? If so, I would suggest setting it to "false" since you're trying to maximize throughput.
In any case, the way I read the code indicates to me that the filtering will be done on the remoting thread servicing the connection from the client. There doesn't appear to be anything inherently single-threaded about it so at this point it looks like there's something about your use-case that is triggering this behavior. Do you have any kind of test you could share which demonstrates this behavior?