8 Replies Latest reply on Jun 4, 2012 11:53 AM by sd_rc

    Messages sitting in queue, not getting picked up by MDB

    rhinmass

      I am having this problem with jboss 6.0.0.Final and hornetq 2.2.5.  No clustering.  I can see the messages in the queue, the queue is not paused and delivery is active (confirmed with JMX)  But the MDBs are not firing.

       

      This was working with 2.1.2, but after upgrading to 2.2.5 the same code exhibits this behavior.

       

      I am stopping and starting delivery to my MDBs programatically, and it seems to freeze up particularly after a stop and start. I've checked in JMX and DeliveryActive is true.

       

      jmx screen.JPG

        • 1. Re: Messages sitting in queue, not getting picked up by MDB
          ataylor

          MDB's are not really part of hornetq, however if you check in the JMX console you can see if the queue has any consumers, if not, i'm guessing the MDB's/RA is configured incorrectly

          • 2. Re: Messages sitting in queue, not getting picked up by MDB
            rhinmass

            Each queue has one consumer.  Same code and configuration is working and in production with hornetq 2.1.2.  I have set up a testbed to use 2.2.5 and haven't made any changes to the code (except for minor change because getMessageCount() now returns long).

             

            Here is the JMX console for one of the queues. 

             

            jmx screen2.JPG

             

             

            One observation is that it tends to happen when I hit the queue with a bunch (20 or more) of messages quickly.  (my messages themselves are small)  If I just send it one or two and let those get processed then open the faucet, it doesn't get in this stuck condition.

             

            Also, when it is in the stuck condition, it doesn't responde to ctrl-c in the window to shut down.  I have to go into task manager to kill it.

             

            I am also seeing a lot of these messages with 2.2.5 that I did not have before.

             

            11:43:48,871 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] Timed out waiting for handler to complete processing

            11:43:58,887 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] Timed out waiting for handler to complete processing

            11:44:08,919 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] Timed out waiting for handler to complete processing

            11:44:18,920 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] Timed out waiting for handler to complete processing

            11:44:28,921 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] Timed out waiting for handler to complete processing

            11:44:38,937 WARN  [org.hornetq.core.client.impl.ClientConsumerImpl] Timed out waiting for handler to complete processing

             

            These seem to happen after I stop and start delivery using:

             

              mBeanServer.invoke(objname, "stopDelivery", new Object[] {}, null);

            and

            mBeanServer.invoke(objname, "startDelivery", new Object[] {}, null);

             

             

            If needed I can provide code to reproduce.

            1 of 1 people found this helpful
            • 3. Re: Messages sitting in queue, not getting picked up by MDB
              ataylor

              Your MDB has a pool size of 15 so there should be 15 consumers on the queue. This means they have been disconnected for some reason, maybe the MDB pool is removing them from the pool because of some error and then thye HornetQ consumer time out the call to onMessage. Are you doing anything in the MDB that could cause it to hang or crash?

              1 of 1 people found this helpful
              • 4. Re: Messages sitting in queue, not getting picked up by MDB
                ataylor

                just as an aside, you get that message when the consumer is being closed whilst in the onmessage, so it looks like the MDB layer is closing the MDB, i would check your MDB code theres probably something wrong there

                • 5. Re: Messages sitting in queue, not getting picked up by MDB
                  rhinmass

                  Thank you Andy.  This is very helpful.  I'm a little confused about the difference between the maxPoolSize and maxSessions.  I want only one MDB instance listening - as too much concurrency gives me database problems - so I have set maxSessions to 1.  I had not changed maxPoolSize, however.  Should it also be set to 1? 

                   

                  I set this in ejb-jar.xml using:

                   

                  <message-driven>

                        <ejb-name>ConfigurableMessageBean0</ejb-name>

                        <ejb-class>com.czen.email.ConfigurableMessageBean</ejb-class>

                        <message-destination-type>javax.jms.Queue</message-destination-type>

                        <activation-config>

                          <activation-config-property>

                            <activation-config-property-name>destination</activation-config-property-name>

                            <activation-config-property-value>Email.0</activation-config-property-value>

                          </activation-config-property>

                          <activation-config-property>

                            <activation-config-property-name>maxSession</activation-config-property-name>

                            <activation-config-property-value>1</activation-config-property-value>

                          </activation-config-property>

                          <activation-config-property>

                            <activation-config-property-name>destinationType</activation-config-property-name>

                            <activation-config-property-value>javax.jms.Queue</activation-config-property-value>

                          </activation-config-property>

                          <activation-config-property>

                            <activation-config-property-name>DeliveryActive</activation-config-property-name>

                            <activation-config-property-value>false</activation-config-property-value>

                          </activation-config-property>

                        </activation-config>

                     </message-driven>

                   

                  (DeliveryActive is set to false, but then I turn it on using mbeanServer once the system has come up.)

                   

                  The idea is that someone can change the # of listeners by editing ejb-jar.xml .  If you could clarify what maxPoolSize does, and whether I need that to match maxSessions or not, I would be grateful.

                  • 6. Re: Messages sitting in queue, not getting picked up by MDB
                    ataylor

                    Thank you Andy.  This is very helpful.  I'm a little confused about the difference between the maxPoolSize and maxSessions.

                    Max sessions is how many HornetQ Sessions feed the MDB endpoint, maxPoolSize is the maximun number of MDB instances there can be. So if you have 1 session feeding 15 MDB you wont get any concurrency as only 1 message can be sent to the endpoint at any one time, its the same in reverse. Best practice is to keep them the same

                    1 of 1 people found this helpful
                    • 7. Re: Messages sitting in queue, not getting picked up by MDB
                      rhinmass

                      interesting ... 

                       

                      My goal is for IT personnel to be able to change the number of listeners in production.  By using maxSessions and putting it in the activatation config in ejb-jar.xml, it was easy enough for them to stop the server, edit ejb-jar.xml, and restart.

                       

                      However , by reading your description, maybe the value to change is maxPoolSize on the MDB rather than maxSessions on the Queue.  And would it be safe to have maxSessions at a larger value than maxPoolSize (because my IT folks don't want to have to change 2 parameters, and worry about whether they agree or not.)

                       

                      I might have to go over to the JBoss discussion group, but do you know how maxPoolSize can be changed without using an annotation, which cannot be changed in the field.

                       

                      Sorry to keep asking more questions. 

                      • 8. Re: Messages sitting in queue, not getting picked up by MDB
                        sd_rc

                        I had a question regarding the maxPoolsize. is that the number of threads that service the incoming messages on the queue?

                        I set the value of maxPoolSize and I still see more than one thread servicing the messages.