7 Replies Latest reply on Dec 7, 2007 9:29 AM by chipschoch

    Delayed Message Sucking

    chipschoch

      JBossSS 4.2.2.GA, JBM 1.4.0.SP1

      Topography:
      2 Node linux cluster running JBM. (L1, L2)
      2 Unclustered windown JBoss servers with services that consume from linux cluster JBM. (W1, W2)

      Using ClusteredConnectionFactory definition on W1, W2 such that both machines are connecting to partial queue on L1. Posting messages to clustered (partial) queues in round-robin fashion such that all messages posted to partial queue L2 are sucked to L1 for consumption by W1, W2.

      Each message posted contains a correlation id that the consumer puts in a response message that is posted to an output queue.

      I post 10 messages and I receive 5 responses. Using a combination of logging and eclipse debugger I have determined that the 5 missing responses are stuck in a loop on L2 in ClientConsumer.getMessage(). Meanwhile, the 5 response messages are sitting in the output queue on L1. (verified by jmx-console). Posting another message causes all 5 messages to then get sucked from L1 to L2 and the processing completes. Also, after some indeterminate amount of time, without posting additional messages, the messages seem to be eventually sucked over from L1 to L2 and processing completes.

      Is there any way to stop these messages from getting hung up on L1. Is there tuning for the MessageSucker.

      Thanks,

        • 1. Re: Delayed Message Sucking
          timfox

          I didn't understand your topology.

          Can you explain it again in more detail?

          • 2. Re: Delayed Message Sucking
            chipschoch

            One other thing. My consumer services are running 5-10 user threads each so they will process 5-10 messages simultaneously.

            • 3. Re: Delayed Message Sucking
              chipschoch

               

              -------------
              | L1 L2 | <-- linux cluster, 2 nodes, with clustered Queues, Q1 and Q2
               -------------
               ^ ^
               | |
               W1 W2
              
              L == linux JBossAS 4.2.2.GA with JBM 1.4.0.SP1
              W == Windows JBossAS 4.2.2.GA. Not clustered , running consumer service
              
              


              Identical services running on L1 and L2 each post request messages to a clustered queue (Q1). The thread that posted a message then calls MessageConsumer.receive() on a response queue (Q2).

              Both W1 and W2 are connected to L1 using ClusteredConnectionFactory. Therefore, all messages posted to L2 are sucked over to L1 and consumed by W1 and W2. W1 and W2 post response messages to response queue (Q2) on L1. The MessageConsumer.receive() called by the L1 service receives the response messages it is waiting for. The MessageConsumer.receive() called by the L2 service never returns. The response messages that L2 is waiting for are stuck on L1-Q2. Eventually, they are sucked over to L2-Q2 and L2 finally receives them.



              • 4. Re: Delayed Message Sucking
                timfox

                 

                "chip_schoch" wrote:
                W1 and W2 post response messages to response queue (Q2) on L1.


                Ok so you've posted your messages to clustered response queue Q2 on L1



                The MessageConsumer.receive() called by the L1 service receives the response messages it is waiting for.


                Yes L1 should receive all the response messages


                The MessageConsumer.receive() called by the L2 service never returns. The response messages that L2 is waiting for are stuck on L1-Q2.


                If the messages are on Q2 on L1 the consumer on L1 should receive them all. A clustered queue *always* favours local consumers. I wouldn't expect the consumer on L2 to receive any of them


                Eventually, they are sucked over to L2-Q2 and L2 finally receives them.


                I guess this happens when you close the consumer on L1?

                If you want your response queue to be only usable by your connection, you can use the distributed request/response pattern.

                I.e. create a clustered temporary queue and pass that as the "replyTo" header in the message.

                When the message is consumed, a reply is sent back to the destination that's specified in that header.

                Sounds like that's what you want to do.



                • 5. Re: Delayed Message Sucking
                  timfox

                  Have a look at org.jboss.test.messaging.jms.clustering.DistributedRequestResponseTest for an example of how to do this.

                  http://anonsvn.jboss.org/repos/messaging/branches/Branch_Stable/tests/src/org/jboss/test/messaging/jms/clustering/DistributedRequestResponseTest.java

                  • 6. Re: Delayed Message Sucking
                    chipschoch



                    If the messages are on Q2 on L1 the consumer on L1 should receive them all. A clustered queue *always* favours local consumers. I wouldn't expect the consumer on L2 to receive any of them


                    Hmm... Sometimes it receives all, sometimes some messages. I have been testing by sending 10 messages at a time, 5 are queued to L1-Q1 and 5 are queued to L2-Q1. Each L server has a service that has a local thread pool in which each thread sends a message then calls MessageConsumer.receive() on Q2. All the response messages are queued to L1-Q2 but only half of them are comsumed by the L1 MessageConsumer.receive() threads. The others get consumed by L2-Q2 MessageConsumer.receive(). In the cases when they are not being consumed there is no consumers on L1-Q2 but 5 consumers on L2-Q2. I assumed they should be getting sucked over to L2-Q2. I wonder why they are sometimes.

                    Could you elaborate on you suggested solution a bit? When you suggest using a clustered temporary queue then aren't I losing my persistence on the response message? Maybe I don't need it?

                    • 7. Re: Delayed Message Sucking
                      chipschoch

                      Tim,

                      Thanks once again for your help. Using the Temporary Queue cleared up all my issues. It works great now.