6 Replies Latest reply on Mar 15, 2012 10:56 AM by swapnil surve

    HornetQ messages lost for slow consumer

    swapnil surve Newbie

      I am using HornetQ running in standalone mode and the client in JBOSS AS 5.1.0

      I am running test case where I am sending 10000 objectmessage with id and value. The consumer is slow consumer which persist the id,value in table and then thread sleeps for 1 second.The connection factory is XA configured.

      During the consumption I am restarting the JBoss. The behaviour I am seeing is on first shutdown, no of messages on queue + no of messages consumed are same as no of messages sent. (I am checking the no of message based on no of id,value pair got persisted in database). On Next restart I see the count is lesser. On subsequent restarts the count is significantly lower. So no of messages persisted are pretty less than no of messages sent. I am loosing few thousand messages. This is unexpected behaviour.

       

       

      One of the log statement says

      2012-03-09 19:39:28,155 DEBUG [org.hornetq.ra.inflow.HornetQActivation] (WorkManager(2)-2) Using context {java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory, java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces:org.jboss.naming:org.jnp.interfaces} for org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@6ab726 destination=queue/slowConsumerQueue destinationType=javax.jms.Queue ack=Auto-acknowledge durable=false clientID=null user=null maxSession=10)

      But I have configured queue to be durable. (its default value)

       

      What can be wrong here? I am attaching relevant files here.

        • 1. Re: HornetQ messages lost for slow consumer
          swapnil surve Newbie

          additional details

          HornetQ version is 2.5.0 final.

          When I restart HornetQ in between (after bringing down the jboss), the no of messages on queue after broker restart are same as them before broker restart.

          • 2. Re: HornetQ messages lost for slow consumer
            Clebert Suconic Master

            That durable on activation would be used only if this was a topic subscription.. which is not the case.

             

             

            Can you try this on AS7? (just as a test)?

             

            Make sure you enable persistence on the standalone.xml.

             

             

            You can also take a look on what PrintData is showing. (If these messages are being ACKed or not):

             

             

            java -Xmx2G -cp /path-to-jars/hornetq-logging.jar:/path-to-jars/hornetq-core.jar:/path-to-jars/netty.jar org.hornetq.core.persistence.impl.journal.PrintData /path-to-journal/bindins /path-to-journal/journal

             

             

             

            This should tell you excatly what you have inside the journal.

             

             

            Also: it was not clear to me if you're using the MDB in-vm or remotely.. it seems you removed the in-vm acceptor.

             

            Take a look also at the consumer-window-size. It seems you need to disable client buffering.

            • 3. Re: HornetQ messages lost for slow consumer
              swapnil surve Newbie

              Thanks for the reply.

              Let me try out with setting consumer window size to 0. I was not aware of PringData. So couldnt find ACK logs. I will check on that.

              I have remote MDB listening to queue. Since HornetQ is running as standalone and JBoss connecting to standalone HornetQ using JCA resource adapter. Is something wrong with the config?

              • 4. Re: HornetQ messages lost for slow consumer
                swapnil surve Newbie

                I tried with consumer window size as 0. Still facing same problem.

                Attaching PrintData logs when JBoss was brought down.

                I set following in ra.xml

                <config-property>

                        <description>The consumer window size</description>

                        <config-property-name>ConsumerWindowSize</config-property-name>

                        <config-property-type>java.lang.Integer</config-property-type>

                        <config-property-value>0</config-property-value>

                      </config-property>

                 

                Also on hornetq-jms.xml following is set.

                <connection-factory name="NettyConnectionFactory">

                      <xa>true</xa>

                      <connectors>

                         <connector-ref connector-name="netty"/>

                      </connectors>

                      <entries>

                         <entry name="/XAConnectionFactory"/>

                      </entries>

                   <consumer-window-size>0</consumer-window-size>

                   </connection-factory>

                 

                 

                This is very abrupt behaviour. Am i missing some configurations? I am attaching PrintData logs.

                • 5. Re: HornetQ messages lost for slow consumer
                  Clebert Suconic Master

                  You didn't post your bindings.. you probably used a wrong directory.

                   

                  it seems something weird on this:

                   

                   

                   

                  ### Message Counts ###

                  message count=16008

                  message reference count

                  queue id 81589,count=7346

                  queue id 7,count=8660

                  prepared message count=0

                   

                   

                  But I need to look at the bindings to what you have.

                   

                   

                  Maybe it would worth to try a checkout of the branch and buildling it?

                  • 6. Re: HornetQ messages lost for slow consumer
                    swapnil surve Newbie

                    Attaching PrintData logs with small set of 100 messages. Even with small set the problem is same. If I am reading the logs correctly, all the messages have been acknowledged. However I didint get them on MDB. There are absolutely no other consumers to this queue.