1 2 Previous Next 20 Replies Latest reply on Mar 3, 2016 9:14 AM by Justin Bertram

    hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

    Eric Zhao Newbie

      Hello,

       

      I got one problem that after the application process run for some days,  the CPU/memory utilization will increase. use jmap dump and found that there are many (>65000) ServerMessageImpl objects existed in memory.  this will cause out of memory after some days.

       

      in our application, we set the ackMode to AUTO ACKNOWLEDGED like the following line:

           topicSession = topicConnection.createSession(transacted,ackMode);

       

      We didn't set the pre-acknowledge in hornetq-jms.xml like what the manual said.

       

      Should we set to pre-acknowledged?

       

      we found normal case, the message can be acknowledged(no ServerMessageImpl object existed in memory). but for unknown cases, the message can't be acknowledged(many serverMessageImpl object existed in memory).

       

      Can someone help me out? what could be the possible reason to cause this issue?

        • 1. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
          Justin Bertram Master

          What value are you using for the transacted variable when you create your session?

          • 2. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
            Eric Zhao Newbie

            Hello Justin,

             

            Thanks for your reply!

            the transacted value is "false".  is it correct for AUTO-ACKNOWLEDGE mode, or I have to use the PRE-ACKNOWLEDGE Mode?

             

            Best Regards,

            Eric.

            • 3. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
              Justin Bertram Master

              is it correct for AUTO-ACKNOWLEDGE mode, or I have to use the PRE-ACKNOWLEDGE Mode?

              Yes, it is proper to use AUTO-ACKNOWLEDGE mode.

               

              Can you provide a reproducible test-case for this scenario?

              • 4. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                Eric Zhao Newbie

                Hello Justin,

                 

                thanks for your confirm, currently we don't know in which case we can reproduce this problem.  we tried to send JMS message in our lab for example 1 message/second in 12 hours. then do jmap, but didn't get the problem. so it could be some error case scneario. since this was a customer site issue, we didn't know the scenario.

                • 5. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                  Justin Bertram Master

                  Message acknowledgment is tested quite thoroughly so in my opinion it's critical to have a clear description of the problematic use-case (ideally in the form of a reproducible test-case).  At this point my guess is that the messages are simply aren't being removed because of some kind of bug in the application (i.e. not in HornetQ).  If you can demonstrate otherwise I'd be happy to investigate further.

                  • 6. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                    Eric Zhao Newbie

                    Justin, sorry for the late response.

                    We trying in our lab in last two weeks, and got the problem again, We suspected when large number of JMS message send to hornetq (and no consumer (PC GUI) connected), is it possible that hornetq store large number of  JMS events and were not released.

                     

                    our TTL is set to 20mins, is it possible TTL hanlding doesn't work, do we have anyway to check it?

                    where the JMS events was stored in hornetq?  how to check and verify if it's the size was full?

                     

                    Thanks in advance for your help.

                     

                    Best Regards,

                    Eric.

                    • 7. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                      Justin Bertram Master

                      our TTL is set to 20mins, is it possible TTL hanlding doesn't work, do we have anyway to check it?

                      If you have an expiry address set then you can look at any queue(s) bound to the expiry address to see the expired messages.  Otherwise you can keep an eye on the message count for the queue(s) in question.

                       

                      where the JMS events was stored in hornetq?

                      I'm not sure what you're asking about the storage of "JMS events."  Can you clarify your question here?

                       

                      how to check and verify if it's the size was full?

                      I'm not clear on this question either.  How do you check if the size of what is full?

                       

                       

                      The bottom line here is...If you're pushing a high volume of messages to a server for 20 minutes with no consumers you're very likely to have memory issues.

                      • 8. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                        Eric Zhao Newbie

                        Dear Justin,

                         

                        We hit the problem on customer labs, finally we got the chance to dump the heap when out of memory.

                         

                        our mx size was set to 2048M, but in heap dump, hornetq occupied almost of all the memory(92.63%), see the attachment.

                         

                        hornetq.PNGhistogram.PNG

                        in hornetq-configuration.xml, we have the following setting:

                           <address-settings>

                              <!--default for catch all-->

                              <address-setting match="#">

                                 <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                 <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                 <redelivery-delay>0</redelivery-delay>

                                 <max-size-bytes>10485760</max-size-bytes>

                                 <message-counter-history-day-limit>10</message-counter-history-day-limit>

                                 <address-full-policy>BLOCK</address-full-policy>

                              </address-setting>

                           </address-settings>

                         

                        how to resolve this kind of issue?  should we use PAGE instead of BLOCK?

                        does the message still existed after expired or in dead letter? the TTL of message is set to 15 mins when publishing the message.

                        • 9. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                          Justin Bertram Master

                          Do you have a test-case that I can use to reproduce this?

                          • 10. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                            Eric Zhao Newbie

                            Justin,

                             

                            I am using JBoss 6.1.0 Final which includes Hornetq 2.2.5 Final version.

                            today we tried the following cases:

                            1). there are client connected to the server. the heap dump has zero number ServerMesageImpl object.

                            2). no client connected to the server. the heap dump has zero number ServerMesasgeImpl object.

                             

                            then I tried to kill the client from task manager after the client connected to the server and found ServerMessageImpl object keep increasing in different heap dump.

                             

                            Do you know which version I can upgrade to fix this issue?

                             

                            Thanks,

                            Eric.

                            • 11. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                              Justin Bertram Master

                              then I tried to kill the client from task manager after the client connected to the server and found ServerMessageImpl object keep increasing in different heap dump.

                              How long did you inspect the server to check the message count?  If a client creates a non-durable subscription and then it gets disconnected for whatever reason (e.g. it crashes, network interruption, etc.) without closing that subscription then that subscription will remain until the TTL for the connection elapses and the server closes the connection and removes the subscription.  Did you inspect the message count on the server both before and after the connection TTL elapsed?

                              • 12. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                                Eric Zhao Newbie

                                Justin,

                                 

                                Thanks for your reply,

                                 

                                in JBOSS6.1.0 Final version, I only find ConnectionTTL in deplooy/jms-ra.rar/META-INF/ra.xml, it's commented out. we tried to set this value into 10min. seems no improvement. the problem still exists.

                                   

                                 

                                      <config-property>

                                        <description>The connection TTL</description>

                                        <config-property-name>ConnectionTTL</config-property-name>

                                        <config-property-type>java.lang.Long</config-property-type>

                                        <config-property-value></config-property-value>

                                      </config-property>

                                 

                                in hornetq-jms.xml. the connection-ttl is -1, that means the connection always there.  but I remembered when we doing the development. it have some problem if it's not -1.

                                 

                                        <connection-factory name="NettyConnectionFactory">

                                                <xa>true</xa>

                                                <connectors>

                                                <connector-ref connector-name="netty"/>

                                                </connectors>

                                                <entries>

                                                        <entry name="/ConnectionFactory"/>

                                                        <entry name="/XAConnectionFactory"/>

                                                </entries>

                                                <connection-ttl>-1</connection-ttl>

                                        </connection-factory>

                                 

                                Is it related with one the above parameter?

                                • 13. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                                  Justin Bertram Master

                                  in hornetq-jms.xml. the connection-ttl is -1, that means the connection always there.  but I remembered when we doing the development. it have some problem if it's not -1.

                                   

                                          <connection-factory name="NettyConnectionFactory">

                                                  <xa>true</xa>

                                                  <connectors>n

                                                  <connector-ref connector-name="netty"/>

                                                  </connectors>

                                                  <entries>

                                                          <entry name="/ConnectionFactory"/>

                                                          <entry name="/XAConnectionFactory"/>

                                                  </entries>

                                                  <connection-ttl>-1</connection-ttl>

                                          </connection-factory>

                                   

                                  Is it related with one the above parameter?

                                  Are the clients in your test using the NettyConnectionFactory?  If so, that would explain the problem.  A connection-ttl of -1 should never be used for connections that could potentially break as it could lead to message accumulation and/or connection accumulation.

                                  • 14. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged
                                    Eric Zhao Newbie

                                    Thanks Justin,

                                     

                                    We changed the connection-ttl into 1hour (3600000ms). it worked with this change.

                                     

                                    One more question:

                                    If No consumer connected, how hornetq handle the server message from Application side, seems it's discarded, right?  since when I dumped, there is no ServerMessageImpl object in heap dump.

                                     

                                    Best Regards,

                                    Eric.

                                    1 2 Previous Next