4 Replies Latest reply on Aug 9, 2012 2:34 AM by stewar

    HornetQ issue : [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception HornetQException[errorCode=101 message=Queue a31de865-ace5-4c78-ac87-a6197a558978 already exists]

    stewar
    stewar Aug 1, 2012 6:07 AM

      Hi,

      I have a setup where Jboss AS 6.1.0 final having HornetQ 2.2.5 final is deployed.

      for a test MDB using a queue, we are regression testing with producer sending 3 JMS messages per second.

      everything works fine till 45000 to 55000 messages are received, takes around 11 hours to be consumed.

       

      later we are continuously seeing the exception in logs :

       

      [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception

      HornetQException[errorCode=101 message=Queue a31de865-ace5-4c78-ac87-a6197a558978 already exists]

       

       

      2012-07-31 13:07:46,977 ERROR [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception

      HornetQException[errorCode=101 message=Queue 7912ef77-ade5-44b8-addc-762f8728e99a already exists]

              at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:1685)

              at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:1028)

              at org.hornetq.core.server.impl.ServerSessionImpl.createQueue(ServerSessionImpl.java:394)

              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:230)

              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)

              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

              at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137)

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

      2012-07-31 13:07:53,722 ERROR [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception

      HornetQException[errorCode=101 message=Queue 6969fc05-eb31-4646-bd5d-10bb127745c7 already exists]

              at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:1685)

              at org.hornetq.core.server.impl.HornetQServerImpl.createQueue(HornetQServerImpl.java:1028)

              at org.hornetq.core.server.impl.ServerSessionImpl.createQueue(ServerSessionImpl.java:394)

              at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:230)

              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)

              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

              at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:137)

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:662)

       

       

       

      and the receiver is reporting : Exception: Producer is closed.

       

      After some hours the jboss server goes down with Out Of Memory Exception :

      2012-08-01 00:57:06,807 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885)) Exception in thread "Thread-10209 (group:HornetQ-client-global-threads-960538885)" java.lang.OutOfMemoryError: unable to create new native thread

      2012-08-01 00:57:06,808 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at java.lang.Thread.start0(Native Method)

      2012-08-01 00:57:06,808 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at java.lang.Thread.start(Thread.java:640)

      2012-08-01 00:57:06,808 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:236)

      2012-08-01 00:57:06,808 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:93)

      2012-08-01 00:57:06,808 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.remoting.impl.invm.InVMAcceptor.disconnect(InVMAcceptor.java:206)

      2012-08-01 00:57:06,809 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener.connectionDestroyed(InVMConnector.java:193)

      2012-08-01 00:57:06,809 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:93)

      2012-08-01 00:57:06,809 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:563)

      2012-08-01 00:57:06,809 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:302)

      2012-08-01 00:57:06,809 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1376)

      2012-08-01 00:57:06,810 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885)) Exception in thread "Thread-10685 (group:HornetQ-client-global-threads-960538885)"        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

      2012-08-01 00:57:06,810 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

      2012-08-01 00:57:06,810 ERROR [STDERR] (Thread-10209 (group:HornetQ-client-global-threads-960538885))   at java.lang.Thread.run(Thread.java:662)

       

      I am using the default configurations for hornetq-configuration.xml, hornetq-jms.xml and ra.xml.

      Can someone let me know what am i missing here? or is there an issue with HornetQ?

        • 1. Re: HornetQ issue : [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception HornetQException[errorCode=101 message=Queue a31de865-ace5-4c78-ac87-a6197a558978 already exists]
          ataylor
          ataylor Aug 1, 2012 6:17 AM (in response to stewar)

          try to fox the OOM issue first, make sure you arent using any anti patterns like creating a connection/session for every message, then go from there.

            • Actions
          • 2. Re: HornetQ issue : [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception HornetQException[errorCode=101 message=Queue a31de865-ace5-4c78-ac87-a6197a558978 already exists]
            stewar
            stewar Aug 1, 2012 6:59 AM (in response to ataylor)

            No,  i am not creating a connection/session for every message,

            we have a singleton class where we create a single connection, session and a producer and use them all along through the lifecycle.

            and this same logic used to work on Jboss 4.2.3 with Jboss Messaging. We have recently migrated to Jboss 6.1.0 and HornetQ 2.2.5 final and facing this issue.

              • Actions
            • 3. Re: HornetQ issue : [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception HornetQException[errorCode=101 message=Queue a31de865-ace5-4c78-ac87-a6197a558978 already exists]
              ataylor
              ataylor Aug 1, 2012 7:17 AM (in response to stewar)

              the java.lang.OutOfMemoryError: unable to create new native thread error implies that your system does not have enough threads available, to be honest without any more info its hard to see what the issue could be. maybe post some test code and configuration we could take a look at.

                • Actions
              • 4. Re: HornetQ issue : [org.hornetq.core.protocol.core.ServerSessionPacketHandler] Caught exception HornetQException[errorCode=101 message=Queue a31de865-ace5-4c78-ac87-a6197a558978 already exists]
                stewar
                stewar Aug 9, 2012 2:34 AM (in response to ataylor)

                I got rid of the OOM Error, now i am getting a very different exception.

                After running the Application for a few hours . I am getting this exception :

                 

                Exception: Timed out waiting for response when sending packet 71.

                 

                I am using default Hornet Configurations file as shipped with JBOSS 6.1.0 final.

                added a few queues to hornetq-jms.xml.

                Queue :

                 

                <queue name="ExampleQueue">

                      <entry name="jms/queue/ExampleQueue"/>

                   </queue>

                 

                 

                Producer Code :

                //the initialise happens only once from the private constructor. its a singleton class basically.

                 

                private void initializeQueue() throws ExampleException {

                                    try {

                                              // 1: Lookup connection factory

                                              factory = (ConnectionFactory) context

                                                                  .lookup("jms/example/connectionFactory");

                 

                 

                                              // 2: Use connection factory to create JMS connection

                                              connection = factory.createConnection();

                 

                 

                                              // 3: Use connection to create a session

                                              session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

                 

                 

                                              // 4: Lookup destination

                                              ExampleQueue = (Queue) context

                                                                  .lookup("jms/queue/ExampleQueue");

                 

                 

                                              // 5: Create a message publisher

                                              sender = session.createProducer(ExampleQueue );

                                              om = session.createObjectMessage();

                 

                 

                                    } catch (Exception e) {

                                              logger.error("Exception during initializing the queue. "

                                                                  + e.getMessage());

                                              throw new ExampleException(

                                                                  "Exception during initializing the queue. "

                                                                                      + e.getMessage());

                                    }

                          }

                 

                //send Event keeps happening on receiving each onMessage() call.

                 

                 

                public void sendEvent(ExampleDTO exampleDTO) {

                                    try {

                                              om.setObject(exampleDTO);

                                              sender.send(om);

                                    } catch (Exception e) {

                                              logger.error("Listener: SendEvent Exception: " + e.getMessage());

                                    }

                                    logger.fine("Listener: SendEvent: Complete");

                          }

                  • Actions