1 2 Previous Next 19 Replies Latest reply on Sep 19, 2011 9:56 PM by clebert.suconic

    Losing STOMP consumer ocasionally

    kesavkolla

      We have a STOMP Websockets client from our UI application, which connects to Hronet topic.  We are using JBoss 6.1.0 and configured the topic like the following:

       

         <topic name="LogTopic">

            <entry name="/topic/LogTopic"/>

         </topic>

       

      We subscribe to this topic and send/receive messages.  Ocasinally we see some of the clients fail to receive messages and I can see the following error in the log file:

       

      2011-09-16 10:55:07,425 WARN  [org.hornetq.core.server.impl.QueueImpl] (Thread-17 (group:HornetQ-server-threads1683676786-1973868001)) removing consumer which did not handle a message, consumer=org.hornetq.core.server.impl.ServerConsumerImpl@66724953, message=Reference[4295367188]:NON-RELIABLE:ServerMessage[messageID=4295367188,priority=4,expiration=0, durable=false, address=jms.topic.MgrTopic,properties=TypedProperties[{PROCESSID=96113, TASKID=C191, DESTVIEW=ENGINEER, CATEGORY=CACHE}]]: java.lang.OutOfMemoryError: Java heap space

              at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:83) [:6.1.0.Final]

              at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:58) [:6.1.0.Final]

              at org.hornetq.core.protocol.stomp.StompSession.sendMessage(StompSession.java:108) [:6.1.0.Final]

              at org.hornetq.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:704) [:6.1.0.Final]

              at org.hornetq.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:291) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2017) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1587) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:6.1.0.Final]

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_24]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_24]

              at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]

       

       

      2011-09-16 10:55:07,426 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-17 (group:HornetQ-server-threads1683676786-1973868001)) Caught unexpected Throwable: java.util.NoSuchElementException

              at org.hornetq.utils.PriorityLinkedListImpl$PriorityLinkedListIterator.repeat(PriorityLinkedListImpl.java:189) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1604) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1472) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:72) [:6.1.0.Final]

              at org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2299) [:6.1.0.Final]

              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_24]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_24]

              at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]

       

      When this error happens the client becomes dead can't receive any further message.  The message itself is very tiny there is no reason to suspect a heap out of space for processing that message.  Our JBoss server functions well under limits of memory no issues there.  Only the STOMP client gets disconnected.

       

      On the client side I can probably detect the connection loss and reconnect again but the message is lost completely. 

       

      Any one can suggest me how to handle these sort of situations?  How to get these messages reliably over STOMP?

       

      Thanks in advance.

        • 1. Re: Losing STOMP consumer ocasionally
          clebert.suconic

          We are implementing Stomp 1.1 as we speak. Stomp 1.0 didn't have any Ping/pong functionality, so if you don't do anything through stomp for a given period, the server will close the connection.

           

           

          You may tweak the acceptor and change the TTL.

           

          We will improve this over the next release where we are doing a major work on Stomp by implementing Stomp 1.1. Howard Gao is doing this work as we speak.

          • 2. Re: Losing STOMP consumer ocasionally
            kesavkolla

            It's not because of timeout.  We've handled timeout in our client code.  We are pinging (sending STOMP messages) for every 1 min to keep the connection alive.

            • 3. Re: Losing STOMP consumer ocasionally
              clebert.suconic

              ok.. I thought you were talking about something different.

               

              If you can produce a simple test replicating this, it would be great.

              • 4. Re: Losing STOMP consumer ocasionally
                kesavkolla

                My main concern is the below: 

                 

                [org.hornetq.core.server.impl.QueueImpl] (Thread-17 (group:HornetQ-server-threads1683676786-1973868001)) removing consumer which did not handle a message, consumer=org.hornetq.core.server.impl.ServerConsumerImpl@66724953, message=Reference[4295367188]:NON-RELIABLE:ServerMessage[messageID=4295367188,priority=4,expiration=0, durable=false, address=jms.topic.MgrTopic,properties=TypedProperties[{PROCESSID=96113, TASKID=C191, DESTVIEW=ENGINEER, CATEGORY=CACHE}]]: java.lang.OutOfMemoryError: Java heap space

                 

                Why is it failing for heap space?  Unfortunately I don't have a fixed mechanism to reproduce this.  This error comes few times in a day, there is no pattern I could predict so far.

                 

                Regards,

                • 5. Re: Losing STOMP consumer ocasionally
                  clebert.suconic

                  Are you sure you are acking your messages properly?

                   

                  Take a look at StompSession's implementation. There's an ACK list that won't be get cleared until you ack your message.

                  • 6. Re: Losing STOMP consumer ocasionally
                    kesavkolla

                    If Ack has an issue it should have been predictable.  I'm using Stomp Websockets from my UI.  As I said most of the time it works but annoyingly I lose some messages in between.

                     

                    The stack trace shows like the following:

                     

                    at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:83) [:6.1.0.Final]

                            at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:58) [:6.1.0.Final]

                            at org.hornetq.core.protocol.stomp.StompSession.sendMessage(StompSession.java:108) [:6.1.0.Final]

                     

                     

                     

                    Not sure what's wrong with that readNullableSimpleString why it throws an error that too intermittently.

                     

                    I'll take a look at StompSession implementation for some clues.

                    • 7. Re: Losing STOMP consumer ocasionally
                      clebert.suconic

                      We will be taking a look on next week.

                      • 8. Re: Losing STOMP consumer ocasionally
                        gaohoward

                        It looks like a potential mem leak issue to me.

                         

                        Can you provide more information on this, like version of hornetq, How often you send and receive stomp messages?

                        Or better if you can give a example client that demonstrate your application.

                         

                        Thanks

                        Howard

                        • 9. Re: Losing STOMP consumer ocasionally
                          kesavkolla

                          I'm using JBoss 6.1.0 that comes with horneq (2.2.5 final).  My application sends and receives messages few hundreds of messages in a span of few hours.  During testing the frequency of messages are little more means those few hundred of messages are happening in 30 min.

                           

                          I'm using Stomp.js (http://jmesnil.net) on the client side in conjuction with Websockets (for IE browsers using flash sockets).  I'm not sure about memory leak as such because after the exception occurs, if I re-establish connection from client things work again normal.  I never had to restart the server.  It's the client which ever got disconnected needs to re connect.

                           

                          My issue is I can re-connect on the client side but the message that was published on the topic will not be redelivered to the client.

                           

                          Thanks

                          -kesav

                          • 10. Re: Losing STOMP consumer ocasionally
                            clebert.suconic

                            Maybe flow control also?

                            • 11. Re: Losing STOMP consumer ocasionally
                              gaohoward

                              Thanks for the info. Looking at the code the OutOfMemory line is

                               

                                byte[] data = new byte[len];

                               

                              The only reason I can think of is that the len happened to be very large when OOM occurred.

                               

                              I'll investigate a bit more.

                               

                              Howard

                              • 12. Re: Losing STOMP consumer ocasionally
                                gaohoward

                                Hi kesav,

                                 

                                I have another question. In your application, when you send messages to HornetQ, what's the type of the messages? TextMessage or other? Are they all of one type of messages, or mixed type of messages?

                                 

                                Thanks

                                Howard

                                • 13. Re: Losing STOMP consumer ocasionally
                                  kesavkolla

                                  The flow control of my application is:  The application is kind of approval process.  Several people be on our web application who are doing several tasks and some tasks are waiting for other people.  The way we used STOMP is when ever there is an activity happens on our application we submit messages.  These messages appear on other people's screen.  When any one need permission from other we send a permission message and other who ever is on the system has correct approval previleges will approve the message.

                                   

                                  The idea of the application is real time collaboration and we don't want to ask every one to refresh their browsers every time.  Currently we are not persisting the approval messages just depending on real time availability.  Our application users are always on the web page either performing tasks or waiting for approvals.  We've implemented background ping to avoid timeouts and also we are sending a keep alive message after every 50 sec to STOMP topic to make sure the connections are not closed.

                                  • 14. Re: Losing STOMP consumer ocasionally
                                    kesavkolla

                                    We are always sending text message no binary type data.

                                    1 2 Previous Next