Load tests failing due to OutOfMemoryError : GC overhead limit exceeded

[mannu_cse]

Hi,

We are doing performance/load testing of messaging in our application and using two JBOSS 7.1.3 node wih hornetq 2.2.23.Final ( each with 1301 MB JVM memory)

Message sender application is deployed in node1 and it sends millions of message on one queue in node1 and receiver is deployed in node2

over the time we see the heap memory growing up and roughly after 90 hours run we see OutOfMemoryError

we have desing 2 queues and 3 topics defined in jms-destination but for this tests we are using only one queue  so other destinations are mostly empty

hornetq configuration is

                <clustered>true</clustered>

                <persistence-enabled>true</persistence-enabled>

                <security-enabled>false</security-enabled>

                <cluster-user>guest</cluster-user>

                <cluster-password>${jboss.messaging.cluster.password:guestp}</cluster-password>

                <journal-type>NIO</journal-type>

                <journal-file-size>102400</journal-file-size>

                <journal-min-files>2</journal-min-files>

address settings is using PAGE mode

                <address-settings>

                    <address-setting match="#">

                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                        <redelivery-delay>0</redelivery-delay>

                        <max-size-bytes>10485760</max-size-bytes>

                        <page-size-bytes>1048576</page-size-bytes>

                        <address-full-policy>PAGE</address-full-policy>

                        <message-counter-history-day-limit>10</message-counter-history-day-limit>

                        <redistribution-delay>0</redistribution-delay>

                    </address-setting>

                </address-settings>

Messages are Non persistent and auto ack

session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

After the crash I took the heap dump and analyzed using eclipse MAT, surprised to see org.hornetq.core.server.impl.QueueImpl taking around 1 GB space

heap dump screen shot is attached.

any help will be greatly appreciated

[clebert.suconic]

Perhaps your test is creating many temporary queues and not closing them?

We had a bug some time ago about leaking queues in certain cases.. but I don't think it's your case.

[mannu_cse]

Hi Clebert,

Thanks for the reply, We are not  using any temporary queue

and we close all session, connection, and producer in finally block for each message sending ( we are testing non- batch messaging)

I am wondering how  can QueueImpl occupy 1 GB. Do you reckon there any other possible reason for this behaviour?

//BR

Manoj

[clebert.suconic]

apparently messages are building up for some reason. you only have 152 queues... you would only get that many messages if messages were building up.. which I'm not sure what's the case now.

Can you try setting persistence=true just as a test.. keep sending your messages as non-persistent.. just set backup persistence=true.

[mannu_cse]

Hi Clebert,

We already have <persistence-enabled>true</persistence-enabled>

Not sure which configuration you are refering to by "set backup persistence=true"

could you plz clarify it.

//BR

Manoj

[mannu_cse]

To make it more clear, on the receiver side we are creating mutiple (50) MessageConsumer, then we dump millions of messages.. since queue size is set to 100MB with PAGING

so if the messages are exceeding this size 100MB then they should be paged and at any point of time queue size in heap should not exceed to 1GB

Is it possible that Session.AUTO_ACKNOWLEDGE is not working and messages remain in the queue even after reception, could this cause the queue to get flooded with messages?

here is the code snippet of the receiver-

@ApplicationScoped

public class JMSMessageObserver implements MessageListener {

      @PostCreate

     public void activate()

     {

          for(int i; i<50; i++)

          {

               final MessageConsumer mc = this.connection.createSession(false, Session.AUTO_ACKNOWLEDGE).createConsumer(destination, filters);

               mc.setMessageListener(this);

               // add mc to a static Map

           }

     }

     @PreDestroy

     public void deactivate()

     {

          for(iterate MessageConsumer  from Map values)

          {

               mc.setMessageListener(null);

               mc.close();

          }

     }

     public void onMessage(final Message message)

     {

            // forward message to CDI observer using ObserverMethod.notify

      }

}

[ataylor]

and we close all session, connection, and producer in finally block for each message sending ( we are testing non- batch messaging)

I would start by re using the connections/sessions and producers, its an anti pattern to do this and may cause you issues.

[mannu_cse]

Hi Andy,

in our application we support both batch and non-batch operation,

but we also want to benchmark non-batch mode and want to ensure that it's stable, ofcourse with some performance overhead, which is perfactly acceptable.

Getting OOM Error is something we consider critical show stopper.

when you say this may cause issues did u mean slow rate or memory problem?

Also if some one can verify what happens if consumer is slow and messages are flooded, I believe PAGING will insure that memory usage is under limit specified in address settings which is 100MB.

BTW we are using JmsXA connection factory, and at receiver side we create fixed number of message consumer (50) so we are not using anti pattern at receiver side atleast.. see the JMSMessageObserver.java I posted in previous my reply

[ataylor]

To make it more clear, on the receiver side we are creating mutiple (50) MessageConsumer, then we dump millions of messages.. since queue size is set to 100MB with PAGING

so if the messages are exceeding this size 100MB then they should be paged and at any point of time queue size in heap should not exceed to 1GB

Is it possible that Session.AUTO_ACKNOWLEDGE is not working and messages remain in the queue even after reception, could this cause the queue to get flooded with messages?

here is the code snippet of the receiver-

@ApplicationScoped

public class JMSMessageObserver implements MessageListener {

      @PostCreate

     public void activate()

     {

          for(int i; i<50; i++)

          {

               final MessageConsumer mc = this.connection.createSession(false, Session.AUTO_ACKNOWLEDGE).createConsumer(destination, filters);

               mc.setMessageListener(this);

               // add mc to a static Map

           }

     }

     @PreDestroy

     public void deactivate()

     {

          for(iterate MessageConsumer  from Map values)

          {

               mc.setMessageListener(null);

               mc.close();

          }

     }

     public void onMessage(final Message message)

     {

            // forward message to CDI observer using ObserverMethod.notify

      }

}

Also creating message listeners in A JEE context is illegal, if you used the JmsXA connection factory then this would probably throw an exception.

[ataylor]

when you say this may cause issues did u mean slow rate or memory problem?

Both, slow rate because of the time taken to create a physical connection and instantaite the resources, less so memory but it will take time for resources to be cleaned up on the server, such as network connections etc. I know you say you want to use a non batched mode, but it is still an anti pattern and you should try not to do it, there is nothing wrong with keeping connections/sessions alive indefinately.

Also if some one can verify what happens if consumer is slow and messages are flooded, I believe PAGING will insure that memory usage is under limit specified in address settings which is 100MB.

yes paging will occur, this has been thoroughly tested, messages are depaged into the queue once enough messages have been acked.

BTW we are using JmsXA connection factory, and at receiver side we create fixed number of message consumer (50) so we are not using anti pattern at receiver side atleast.. see the JMSMessageObserver.java I posted in previous my reply

ok in that casse you are ok creating connections every time as they are pooled. I am surprised setMessageListener doesnt throw an exception tho? have you disabled the strict check?

[mannu_cse]

Both, slow rate because of the time taken to create a physical connection and instantaite the resources, ...

       Agreed that there may be some memory used by session/ message producers, But in the heap dump it clearly shows the QueueImpl reaching 1 GB which is very large, plz check the screen shot I have attached

       at the begining of this thread

I am surprised setMessageListener doesnt throw an exception tho? have you disabled the strict check?

      The code is stipped out example snippet, actually setMessageListener() is inside try {....} catch (JMSException e) block

     We know that CDI notifier that we used inside onMessage is very slow so we used multiple message consumer to make little faster, but still its a bottleneck.

     public void onMessage(final Message message)

     {

           Object object = extract(message);

            for (final ObserverMethod<? super Object> observer : observers) {

                    observer.notify(object );  // Very slow

            }

      }

[ataylor]

    Agreed that there may be some memory used by session/ message producers, But in the heap dump it clearly shows the QueueImpl reaching 1 GB which is very large, plz check the screen shot I have attached

       at the begining of this thread

Agreed, I have looked at the dump and yes, there are messages there, Im just saying if it is a bug it could be because you are using this anti pattern,  if you reuse the connections does this still occur?

The code is stipped out example snippet, actually setMessageListener() is inside try {....} catch (JMSException e) block

I mean setMessageListener should throw an IllegalStateException if you are using the JmsXA, we do this because it is illegal in the JEE spec to do anything asynchronous as you dont know what effects will happen?

We know that CDI notifier that we used inside onMessage is very slow so we used multiple message consumer to make little faster, but still its a bottleneck.

have you tried using MDB's?

[mannu_cse]

I mean setMessageListener should throw an IllegalStateException if you are using the JmsXA

       At the receiver side we are not using any anti pattern, we keep one connection open ( which is non XA ) and create fixed number of consumers and pool them, thats why not getting IllegalStateException.

Im just saying if it is a bug it could be because you are using this anti pattern

       I will try batch mode also, but in this scenario we have to support non-batch messaging when multiple apps will use our messaging app jar to send messages in non-batch mode and we want to make sure it is stable in case of message flooding.

      we dont want to use MDB as it's a core jar not an ear and at the moment slow performance is acceptable as long as it's stable, currently it goes OOM Error in 90 hours run.

      anti pattern is unavoidable in this particular case of functionality.     

[ataylor]

  At the receiver side we are not using any anti pattern, we keep one connection open ( which is non XA ) and create fixed number of consumers and pool them, thats why not getting IllegalStateException.

I think you are misunderstanding what I am saying, you should not be setting a message listener in a JEE environment, if you do this using the JmsXA connection factory it should thorw an exception, if you are using the normal connectionfactory it will let you only because you are by passing the JCA managed connection factories but you still shouldn't do it, the reason you shouldnt is that all threads should be in control of the Application Server and by [passing this can lead to weird stuff happening. this is what MDB's are there for, message listeners whos thrtewads are controlled by the AS.

also, but in this scenario we have to support non-batch messaging when multiple apps will use our messaging app jar to send messages in non-batch mode and we want to make sure it is stable in case of message flooding.

like you say, you are using JmsXA so they are pooled so it isnt a problem, you arent actually reopening any connections, they are kept open by the AS, so to be honest, testing batched or non batched shouldnt make any difference apart from creating the producer.

we dont want to use MDB as it's a core jar not an ear

You can deploy MDB's in jars, you dont need an ear any more in AS7 and like I say you should use them instead of message listeners as the AS controls the threads. If you cant use them at leasttry and see what happens, if you get no OOM's then we know its because of this.

[mannu_cse]

Thanks for clarifying Andy.

I'll try with MDB, looks like setMessageListner is not reliable, but its really weird to get OOM

anyways the tests takes ~90 hours to see any failure, will give a try with MDBs for few days and update you about this.