3 Replies Latest reply on Apr 2, 2012 5:58 AM by ayush.vatsyayan

    Consumer / queue association lost

    philipbauwens

      Application deployed in  JBoss 5.1, Queues are configured declaratively.

       

      Incoming requests are converted to an internal structure and subsequently put on a JBoss queue.

      The consumer of the queue picks up the message and processes the request.

       

      To implement the producer\consumer approach, the org.jboss.ejb3.annotation.Producer and org.jboss.ejb3.annotation.Consumer APIs have been used.

      The connection to queue is established as follows.

       

      ProducerManager manager = ((ProducerObject) context.lookup("QueueProducer")).getProducerManager();

      manager.connect();

       

      During this week, on a production system, all of a sudden all the queues started showing unexpected behavior.

      There was no message put on the queue anymore.  In the application, there was no exception seen, although exception handling is implemented.

      After few analysis from JMX and Admin consoles, it was found that the Consumer count for each queue has been changed to 0.

       

      Has anyone seen similar behaviour where consumer association got lost somehow ?

      What could be causing this issue  and why are there no exceptions thrown in case we try to put something on the queues ?

       

      Tia,

      Philip

        • 1. Re: Consumer / queue association lost
          ayush.vatsyayan

          Philip,

           

          With the available information there are 3 things which strikes me

          1) Is producer producing the messages?

          2) Is logging set to debug mode?

          3) Is exception handled in your code and is printing the exception in logs?

           

          Could you please confirm the above things because I believe that if there is any problem then exception will thrown in any case?

          • 2. Re: Consumer / queue association lost
            philipbauwens

            Hi Ayush,

             

            Initially (the time the consumers association was lost) the system was running in WARN level.

            However, as the problem was persistent, we were able to change the loglevel to debug.

             

            The debug logging showed that the message was indeed produced by the producer.

            In our code, exception handling is present, but no exception was seen in the logs.

             

            We think that a throwable was thrown in stead of an exception and these are not handled in our code.

            But this is just a guess.

             

            Our major concernes on this topic are :

            - the fact that we don't see an exception on which we can act.

            - what could be the root cause for the association to get lost.

            • 3. Re: Consumer / queue association lost
              ayush.vatsyayan

              Philip,

               

              If throwable was thrown then also it should print the exception in logs until and unless it's handled somewhere in class hierarchy and not thrown.

              I suggest you to check your code and see if somewhere in class hierarchy the exception is handled because without knowing the exception you cannot solve it 

               

              Below are two general cases which might help you

               

              1) There can be a possibility of High Availability fail over. In HornetQ, fail over or rollback triggers an exception. Your JBoss Messaging application must capture this exception and retry sending the message for it to be compatible with HornetQ. Your JBoss Messaging application must capture this exception and retry sending the message for it to be compatible with HornetQ. use following try block when sending messages

               

              try {

                producer.send(createMessage(session, i));

                System.out.println("Message: " + i);

              } catch (Exception e) {

                  // log error message here just for your debug purpose, not required in deployment package

                   e.printStackTrace();

                Thread.sleep(1000);

                producer.send(createMessage(session, i));

              }

               

               

              2) There can also be a possibility of client detecting a dead connection and terminating it, as it pings the server to detect if the server or network has failed.

              f the client does not receive any packets for client-failure-check-period milliseconds then it will consider the connection failed and will either initiate fail-over, or call any SessionFailureListener instances (or ExceptionListener instances if you are using JMS) depending on how it has been configured.

               

              For JMS, the check period is defined by the ClientFailureCheckPeriod attribute on a HornetQConnectionFactory instance. If JMS connection factory instances are being deployed directly into JNDI on the server side, it can be specified in the JBOSS_DIST/jboss-as/server/PROFILE/deploy/hornetq/hornetq-jms.xml configuration file, using the parameter client-failure-check-period.

               

              The default value for the client failure check period is 30000ms (30 seconds). A value of -1 means the client will never fail the connection on the client side if no data is received from the server.