1 2 Previous Next 17 Replies Latest reply on Mar 21, 2013 11:32 AM by Manoj Uikey

    Load tests failing due to OutOfMemoryError : GC overhead limit exceeded

    Manoj Uikey Newbie

      Hi,

       

      We are doing performance/load testing of messaging in our application and using two JBOSS 7.1.3 node wih hornetq 2.2.23.Final ( each with 1301 MB JVM memory)

      Message sender application is deployed in node1 and it sends millions of message on one queue in node1 and receiver is deployed in node2

      over the time we see the heap memory growing up and roughly after 90 hours run we see OutOfMemoryError

      we have desing 2 queues and 3 topics defined in jms-destination but for this tests we are using only one queue  so other destinations are mostly empty

       

      hornetq configuration is

       

                      <clustered>true</clustered>

                      <persistence-enabled>true</persistence-enabled>

                      <security-enabled>false</security-enabled>

                      <cluster-user>guest</cluster-user>

                      <cluster-password>${jboss.messaging.cluster.password:guestp}</cluster-password>

                      <journal-type>NIO</journal-type>

                      <journal-file-size>102400</journal-file-size>

                      <journal-min-files>2</journal-min-files>

       

      address settings is using PAGE mode

                      <address-settings>

                          <address-setting match="#">

                              <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                              <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                              <redelivery-delay>0</redelivery-delay>

                              <max-size-bytes>10485760</max-size-bytes>

                              <page-size-bytes>1048576</page-size-bytes>

                              <address-full-policy>PAGE</address-full-policy>

                              <message-counter-history-day-limit>10</message-counter-history-day-limit>

                              <redistribution-delay>0</redistribution-delay>

                          </address-setting>

                      </address-settings>

       

      Messages are Non persistent and auto ack

      session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

       

      After the crash I took the heap dump and analyzed using eclipse MAT, surprised to see org.hornetq.core.server.impl.QueueImpl taking around 1 GB space

      heap dump screen shot is attached.

      any help will be greatly appreciated

        • 1. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
          Clebert Suconic Master

          Perhaps your test is creating many temporary queues and not closing them?

           

           

          We had a bug some time ago about leaking queues in certain cases.. but I don't think it's your case.

          • 2. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
            Manoj Uikey Newbie

            Hi Clebert,

             

            Thanks for the reply, We are not  using any temporary queue

            and we close all session, connection, and producer in finally block for each message sending ( we are testing non- batch messaging)

            I am wondering how  can QueueImpl occupy 1 GB. Do you reckon there any other possible reason for this behaviour?

             

            //BR

            Manoj

            • 3. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
              Clebert Suconic Master

              apparently messages are building up for some reason. you only have 152 queues... you would only get that many messages if messages were building up.. which I'm not sure what's the case now.

               

               

              Can you try setting persistence=true just as a test.. keep sending your messages as non-persistent.. just set backup persistence=true.

              • 4. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                Manoj Uikey Newbie

                Hi Clebert,

                 

                We already have <persistence-enabled>true</persistence-enabled>

                Not sure which configuration you are refering to by "set backup persistence=true"

                could you plz clarify it.

                 

                //BR

                Manoj

                • 5. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                  Manoj Uikey Newbie

                  To make it more clear, on the receiver side we are creating mutiple (50) MessageConsumer, then we dump millions of messages.. since queue size is set to 100MB with PAGING

                  so if the messages are exceeding this size 100MB then they should be paged and at any point of time queue size in heap should not exceed to 1GB

                  Is it possible that Session.AUTO_ACKNOWLEDGE is not working and messages remain in the queue even after reception, could this cause the queue to get flooded with messages?

                   

                  here is the code snippet of the receiver-

                   

                  @ApplicationScoped

                  public class JMSMessageObserver implements MessageListener {

                        @PostCreate

                       public void activate()

                       {

                            for(int i; i<50; i++)

                            {

                                 final MessageConsumer mc = this.connection.createSession(false, Session.AUTO_ACKNOWLEDGE).createConsumer(destination, filters);

                                 mc.setMessageListener(this);

                                 // add mc to a static Map

                             }

                       }

                     

                       @PreDestroy

                       public void deactivate()

                       {

                            for(iterate MessageConsumer  from Map values)

                            {

                                 mc.setMessageListener(null);

                                 mc.close();

                   

                            }

                       }

                   

                       public void onMessage(final Message message)

                       {

                              // forward message to CDI observer using ObserverMethod.notify

                        }

                  }

                  • 6. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                    Andy Taylor Master

                    and we close all session, connection, and producer in finally block for each message sending ( we are testing non- batch messaging)

                    I would start by re using the connections/sessions and producers, its an anti pattern to do this and may cause you issues.

                    • 7. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                      Manoj Uikey Newbie

                      Hi Andy,

                       

                      in our application we support both batch and non-batch operation,

                      but we also want to benchmark non-batch mode and want to ensure that it's stable, ofcourse with some performance overhead, which is perfactly acceptable.

                       

                      Getting OOM Error is something we consider critical show stopper.

                      when you say this may cause issues did u mean slow rate or memory problem?

                       

                      Also if some one can verify what happens if consumer is slow and messages are flooded, I believe PAGING will insure that memory usage is under limit specified in address settings which is 100MB.

                       

                      BTW we are using JmsXA connection factory, and at receiver side we create fixed number of message consumer (50) so we are not using anti pattern at receiver side atleast.. see the JMSMessageObserver.java I posted in previous my reply

                      • 8. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                        Andy Taylor Master

                        To make it more clear, on the receiver side we are creating mutiple (50) MessageConsumer, then we dump millions of messages.. since queue size is set to 100MB with PAGING

                        so if the messages are exceeding this size 100MB then they should be paged and at any point of time queue size in heap should not exceed to 1GB

                        Is it possible that Session.AUTO_ACKNOWLEDGE is not working and messages remain in the queue even after reception, could this cause the queue to get flooded with messages?

                         

                        here is the code snippet of the receiver-

                         

                        @ApplicationScoped

                        public class JMSMessageObserver implements MessageListener {

                              @PostCreate

                             public void activate()

                             {

                                  for(int i; i<50; i++)

                                  {

                                       final MessageConsumer mc = this.connection.createSession(false, Session.AUTO_ACKNOWLEDGE).createConsumer(destination, filters);

                                       mc.setMessageListener(this);

                                       // add mc to a static Map

                                   }

                             }

                           

                             @PreDestroy

                             public void deactivate()

                             {

                                  for(iterate MessageConsumer  from Map values)

                                  {

                                       mc.setMessageListener(null);

                                       mc.close();

                         

                                  }

                             }

                         

                             public void onMessage(final Message message)

                             {

                                    // forward message to CDI observer using ObserverMethod.notify

                              }

                        }

                        Also creating message listeners in A JEE context is illegal, if you used the JmsXA connection factory then this would probably throw an exception.

                        • 9. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                          Andy Taylor Master

                          when you say this may cause issues did u mean slow rate or memory problem?

                          Both, slow rate because of the time taken to create a physical connection and instantaite the resources, less so memory but it will take time for resources to be cleaned up on the server, such as network connections etc. I know you say you want to use a non batched mode, but it is still an anti pattern and you should try not to do it, there is nothing wrong with keeping connections/sessions alive indefinately.

                           

                          Also if some one can verify what happens if consumer is slow and messages are flooded, I believe PAGING will insure that memory usage is under limit specified in address settings which is 100MB.

                          yes paging will occur, this has been thoroughly tested, messages are depaged into the queue once enough messages have been acked.

                           

                          BTW we are using JmsXA connection factory, and at receiver side we create fixed number of message consumer (50) so we are not using anti pattern at receiver side atleast.. see the JMSMessageObserver.java I posted in previous my reply

                          ok in that casse you are ok creating connections every time as they are pooled. I am surprised setMessageListener doesnt throw an exception tho? have you disabled the strict check?

                          • 10. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                            Manoj Uikey Newbie

                            Both, slow rate because of the time taken to create a physical connection and instantaite the resources, ...

                                   Agreed that there may be some memory used by session/ message producers, But in the heap dump it clearly shows the QueueImpl reaching 1 GB which is very large, plz check the screen shot I have attached

                                   at the begining of this thread

                             

                            I am surprised setMessageListener doesnt throw an exception tho? have you disabled the strict check?

                                  The code is stipped out example snippet, actually setMessageListener() is inside try {....} catch (JMSException e) block

                             

                                 We know that CDI notifier that we used inside onMessage is very slow so we used multiple message consumer to make little faster, but still its a bottleneck.

                             

                                 public void onMessage(final Message message)

                                 {

                                       Object object = extract(message);

                                   

                                        for (final ObserverMethod<? super Object> observer : observers) {

                                                observer.notify(object );  // Very slow

                                        }

                                  }

                            • 11. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                              Andy Taylor Master

                                  Agreed that there may be some memory used by session/ message producers, But in the heap dump it clearly shows the QueueImpl reaching 1 GB which is very large, plz check the screen shot I have attached

                                     at the begining of this thread

                              Agreed, I have looked at the dump and yes, there are messages there, Im just saying if it is a bug it could be because you are using this anti pattern,  if you reuse the connections does this still occur?

                               

                              The code is stipped out example snippet, actually setMessageListener() is inside try {....} catch (JMSException e) block

                              I mean setMessageListener should throw an IllegalStateException if you are using the JmsXA, we do this because it is illegal in the JEE spec to do anything asynchronous as you dont know what effects will happen?

                               

                              We know that CDI notifier that we used inside onMessage is very slow so we used multiple message consumer to make little faster, but still its a bottleneck.

                              have you tried using MDB's?

                              • 12. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                                Manoj Uikey Newbie

                                I mean setMessageListener should throw an IllegalStateException if you are using the JmsXA

                                       At the receiver side we are not using any anti pattern, we keep one connection open ( which is non XA ) and create fixed number of consumers and pool them, thats why not getting IllegalStateException.

                                 

                                Im just saying if it is a bug it could be because you are using this anti pattern

                                       I will try batch mode also, but in this scenario we have to support non-batch messaging when multiple apps will use our messaging app jar to send messages in non-batch mode and we want to make sure it is stable in case of message flooding.

                                      we dont want to use MDB as it's a core jar not an ear and at the moment slow performance is acceptable as long as it's stable, currently it goes OOM Error in 90 hours run.

                                      anti pattern is unavoidable in this particular case of functionality.     

                                • 13. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                                  Andy Taylor Master

                                    At the receiver side we are not using any anti pattern, we keep one connection open ( which is non XA ) and create fixed number of consumers and pool them, thats why not getting IllegalStateException.

                                  I think you are misunderstanding what I am saying, you should not be setting a message listener in a JEE environment, if you do this using the JmsXA connection factory it should thorw an exception, if you are using the normal connectionfactory it will let you only because you are by passing the JCA managed connection factories but you still shouldn't do it, the reason you shouldnt is that all threads should be in control of the Application Server and by [passing this can lead to weird stuff happening. this is what MDB's are there for, message listeners whos thrtewads are controlled by the AS.

                                  also, but in this scenario we have to support non-batch messaging when multiple apps will use our messaging app jar to send messages in non-batch mode and we want to make sure it is stable in case of message flooding.

                                  like you say, you are using JmsXA so they are pooled so it isnt a problem, you arent actually reopening any connections, they are kept open by the AS, so to be honest, testing batched or non batched shouldnt make any difference apart from creating the producer.

                                  we dont want to use MDB as it's a core jar not an ear

                                  You can deploy MDB's in jars, you dont need an ear any more in AS7 and like I say you should use them instead of message listeners as the AS controls the threads. If you cant use them at leasttry and see what happens, if you get no OOM's then we know its because of this.

                                  1 of 1 people found this helpful
                                  • 14. Re: Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
                                    Manoj Uikey Newbie

                                    Thanks for clarifying Andy.

                                    I'll try with MDB, looks like setMessageListner is not reliable, but its really weird to get OOM

                                    anyways the tests takes ~90 hours to see any failure, will give a try with MDBs for few days and update you about this.

                                    1 2 Previous Next