2 Replies Latest reply on Jun 14, 2012 11:37 PM by sudeeppillai

    JBoss Messaging cluster : Some messages takes 2 to 6 seconds to deliver

    sudeeppillai

      We use JBoss EAP version 5.1.1 for JBoss messaging. JBoss 2 node cluster runs on linux servers and message sending server runs on a windows 2003 server and messaging consumers run on a separate windows 2003 server. Message consumers are not overloaded, yet if there are a constant flow of incoming messages, some messages are delivered after a huge delay and rest of the messages are received in few milli seconds.

       

      I have been working with Redhat support with a sample application to reproduce the scenario, but finally it is being doubted as the network configuration on windows as the reason for this issue. Would like to get some thoughts from experts in this forum and also see if anyone else faced this issue.

       

      Thanks,
      Sudeep

        • 1. Re: JBoss Messaging cluster : Some messages takes 2 to 6 seconds to deliver
          sudeeppillai

          This is fixed by minor configuration change.

           

          In deploy\messaging\remoting-bisocket-service.xml  add the following configuration

            <attribute name="enableTcpNoDelay" isParam="true">true</attribute>

           

          Above configuration had improved the performance.

           

          Since we are using tcp channel instead of udp, we also modified

          \deploy\cluster\jgroups-channelfactory.sar\META-INF\jgroups-channelfactory-stacks.xml

          by adding below configuration to tcp and jbm-data channels

          tcp_nodelay="true"

           

          Both the above configurations helped the message delivery happening in sub-milliseconds time for the sample application which was consistently producing nearly 200ms time delay for 5% of messages.

           

          I would like to hear from expert on any side effects of above configuration especially since Redhat has not documented the use of "enableTcpNoDelay" in JBoss 5.x version documentation.

           

          Thanks,
          Sudeep

          • 2. Re: JBoss Messaging cluster : Some messages takes 2 to 6 seconds to deliver
            sudeeppillai

            Even with enableTcpNoDelay flag set, we experienced high delayed messages. Finally, root cause was identified. We have multiple Consumers for single message queue listening to Messages in its own threads(one thread per consumer). When a consumer receives a message, its thread process that message and at the end of processing calls consumer.receive(timeout) and waits for the next message. Processing time of a message in consumer threads can vary with some message processing taking few seconds.

             

            Problem is JBoss will stream message to the above consumer, which has not called receive method, even though there are other consumers which have called receive method and waiting. If this scenario happens, second message streamed to that consumer will be processed once the consumer thread is free. We avoided this by closing the consumer soon after receive returns a message. When the thread is ready for the next message, create a new consumer and call receive on the new consumer.

             

            Hope this helps.