This is fixed by minor configuration change.

In deploy\messaging\remoting-bisocket-service.xml  add the following configuration

  <attribute name="enableTcpNoDelay" isParam="true">true</attribute>

Above configuration had improved the performance.

Since we are using tcp channel instead of udp, we also modified

\deploy\cluster\jgroups-channelfactory.sar\META-INF\jgroups-channelfactory-stacks.xml

by adding below configuration to tcp and jbm-data channels

tcp_nodelay="true"

Both the above configurations helped the message delivery happening in sub-milliseconds time for the sample application which was consistently producing nearly 200ms time delay for 5% of messages.

I would like to hear from expert on any side effects of above configuration especially since Redhat has not documented the use of "enableTcpNoDelay" in JBoss 5.x version documentation.

Thanks,
Sudeep

Even with enableTcpNoDelay flag set, we experienced high delayed messages. Finally, root cause was identified. We have multiple Consumers for single message queue listening to Messages in its own threads(one thread per consumer). When a consumer receives a message, its thread process that message and at the end of processing calls consumer.receive(timeout) and waits for the next message. Processing time of a message in consumer threads can vary with some message processing taking few seconds.

Problem is JBoss will stream message to the above consumer, which has not called receive method, even though there are other consumers which have called receive method and waiting. If this scenario happens, second message streamed to that consumer will be processed once the consumer thread is free. We avoided this by closing the consumer soon after receive returns a message. When the thread is ready for the next message, create a new consumer and call receive on the new consumer.

Hope this helps.