How you are sending? Non Transactioned / Transactioned?
If you are sending non-transactioned. Each send on the thread will need to wait a roundtrip to the disk before you can send another message.
This is because we send persistent message synchronously. For every send you make, you will have to wait the TimedBuffer kick in, send to the disk and answer that message.
If you want to scale it better, you would need to either use transactions (batched of N elements) or multiple producers, as they will all share the same buffers before it's sent to disk.
BTW: you can make the system to send persistent asynchronously. to not block. It's a simple setting. But you will lose cofirmation that the message was accepted. Maybe that's ok for your business?
Yes, just checking the settings, it is set to not block on durable send and also set to not block on acknowledge. So I suppose that makes my question somewhat non-relevant (sorry). The strange thing about it is that there appears to be no lock contention. Thanks for the reply, I'll have to dig into it a bit more to see where the time is being spent.
edit: Yes, using multiple producers - there are enough in the producer pool for each web service thread to obtain one exclusively.
>> "The strange thing about it is that there appears to be no lock contention." <<<
there shouldn't be...
your client will be blocking until the send was finished.
Everything on HornetQ is asynchronous. You shouldn't have any tasks waiting the block on the disk. You will have a callback on the the TimedBuffer (and AsynchronousFileImpl if using AIO). It all happens through OperationContext.
If you configured the system to block on persistent sends.. your client will block on while sending,and it won't be unblocked until the write is sync on the disk.
It all depends on what you need for your system. We always deliver it configured to be the safest possible option.
What settings did you change?
You should change the connection factory setting.