Load tests failing due to OutOfMemoryError : GC overhead limit exceeded
mannu_cse Mar 6, 2013 10:17 AMHi,
We are doing performance/load testing of messaging in our application and using two JBOSS 7.1.3 node wih hornetq 2.2.23.Final ( each with 1301 MB JVM memory)
Message sender application is deployed in node1 and it sends millions of message on one queue in node1 and receiver is deployed in node2
over the time we see the heap memory growing up and roughly after 90 hours run we see OutOfMemoryError
we have desing 2 queues and 3 topics defined in jms-destination but for this tests we are using only one queue so other destinations are mostly empty
hornetq configuration is
<clustered>true</clustered>
<persistence-enabled>true</persistence-enabled>
<security-enabled>false</security-enabled>
<cluster-user>guest</cluster-user>
<cluster-password>${jboss.messaging.cluster.password:guestp}</cluster-password>
<journal-type>NIO</journal-type>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
address settings is using PAGE mode
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<page-size-bytes>1048576</page-size-bytes>
<address-full-policy>PAGE</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>0</redistribution-delay>
</address-setting>
</address-settings>
Messages are Non persistent and auto ack
session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
After the crash I took the heap dump and analyzed using eclipse MAT, surprised to see org.hornetq.core.server.impl.QueueImpl taking around 1 GB space
heap dump screen shot is attached.
any help will be greatly appreciated
-
heap-dump.sceen.jpg 262.7 KB