14 Replies Latest reply on Apr 14, 2014 5:15 PM by jbertram

While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jcrisp Apr 8, 2014 5:50 PM

Hello!

We are running JBoss EAP 6.2.2 in Domain mode. We have two load balanced Web Service JBoss servers that JMS remote call two JBoss HornetQ clustered servers. Using SoapUI, we load test simulating 10 clients hitting the web service cluster with 1 request each second. The consumer process takes on average 122 milliseconds to complete (so messages are not getting backed up). Within 45 minutes the JVM Heap usage for both JMS nodes at times are at 100% and they begin to fail.

Through a memory dump we noticed that 80% of the classes were of DuplicateIDCacheImpl.java (https://github.com/hornetq/hornetq/blob/master/hornetq-server/src/main/java/org/hornetq/core/postoffice/impl/DuplicateIDCacheImpl.java). We noticed that if we set use-duplicate-detection to false and manually set the duplicate detection id property through code (as defined here: http://docs.jboss.org/hornetq/2.3.0.Final/docs/user-manual/html/duplicate-detection.html) we do not run into this problem:

requestMsg.setStringProperty(org.hornetq.api.core.Message.HDR_DUPLICATE_DETECTION_ID.toString(),UUID.randomUUID().toString());

We have been running 4+ hours straight at this pace so far without issue. The JVM heap is behaving (currently between 24% and 38%)

Can someone verify this is a viable workaround? That the duplicate detection value is not being ignored and we are still getting the Duplicate Message Detection feature?

1. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 8, 2014 7:15 PM (in response to jcrisp)

I'm not aware of any memory leaks around duplicate detection even on earlier versions...

When you use duplciate detection you will have up to your cache size on a hashMap.. so you are expected to have a bunch of strings there depending on your size.

Are you sure you are committing your messages and everything? What versio nare you using?
Actions
2. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jcrisp Apr 9, 2014 12:16 PM (in response to clebert.suconic)

After 24 hours we are at 65%-96% (900,000 messages completed). So there is still some heap growth we need to address.

We are using version 2.3.12 of HornetQ and not running in transactions. We are using temporary queues (request/response) that I should of mentioned as it might be the culprit.

You brought up changing the id-cache-size. The default value is 2000 elements. Do you have any suggestions for our scenario? Since the replyTo temporary queue is responded to in milliseconds should we instead reduce this number?
Actions
3. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jbertram Apr 9, 2014 1:55 PM (in response to jcrisp)

Do you have a simple test we could use to reproduce this? I'm sure you could speed this up quite a bit by sending messages at maximum speed rather than 1 msg/sec.
Actions
4. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 9, 2014 6:41 PM (in response to jcrisp)

The Cache is done per Address. since you are using temporary queue and duplicate IDs... perhaps there's something broken around that.
Why would you need duplicate IDs with temp queues anyways? The queue won't survive a restart anyways.. so DuplicateDetection over TempQueue is an overkil.
Actions
5. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 9, 2014 6:43 PM (in response to clebert.suconic)

Just found a leak for your scenario... create the queue.. use duplicate detection... delete the queue... the duplicate detection will leave forever on PostOfficeImpl. That matches with what you're describing.

I wouldn't recommend use DuplicateDetection with Temp Queues whatsoever. (not just for this bug condition.. it's not expected to happen anyways).
Actions
6. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jcrisp Apr 10, 2014 12:23 PM (in response to clebert.suconic)

Makes sense, thanks Clebert! Based on your suggestion I ran another stress test overnight to verify with the Duplicate Message Detection feature disabled. We look better (min 64%, 75% max after running 800,000 requests). There still appears to be some JVM leak suspects in HornetQ. I did a memory dump and Eclipse MAT report. It reported:

One instance of "org.hornetq.core.settings.impl.HierarchicalObjectRepository" loaded by "org.jboss.modules.ModuleClassLoader @ 0xd11c9f98" occupies 52,683,488 (12.85%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".
One instance of "org.hornetq.core.settings.impl.HierarchicalObjectRepository" loaded by "org.jboss.modules.ModuleClassLoader @ 0xd11c9f98" occupies 52,683,488 (12.85%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class loader>".
785,848 instances of "java.lang.String", loaded by "<system class loader>" occupy 112,519,056 (27.44%) bytes. These instances are referenced from one instance of "org.hornetq.core.remoting.server.impl.RemotingServiceImpl", loaded by "org.jboss.modules.ModuleClassLoader @ 0xd11c9f98"
2 instances of "org.hornetq.core.protocol.stomp.StompProtocolManager", loaded by "org.jboss.modules.ModuleClassLoader @ 0xd11c9f98" occupy 57,844,320 (14.11%) bytes.
The problem suspects 1, 2, 3 and 4 may be related, because the reference chains to them have a common beginning.
Actions
7. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 10, 2014 3:58 PM (in response to jcrisp)

Wow, you must be really creating lots of temporary queues to be hitting this...

When we route, we will go through the address settings for the current setting on a given address. On this case your address is volatile as it's an temp queue.

The routing will get the setting on HierarchicalObjectRepository and will keep a cache there.

Do me a favour to confirm this... Create and Delete *any* queue. Queue deletion should call clearCache what will confirm my suspicious.

Now to the fix: You could open a JIRA.. but since you have an EAP contract the best would be to raise a support ticket and ask a SEG - Support Engineer) can help you get a back port of the fix on the latest available 2.3.x version for your branch... I'm not very familiar with the support inner works .. they would help you better on this.

But since you came through the open source channel, can you also open the JIRA for "Memory Leaks with Temporary Queues"... and you could open another one for "memory leaks with Temporary Queues and Duplicate Detection"... well. .they are sort of related... if you prefer you could open a single one containing all that information.

if you prefer not to open the JIRA let me know and we will handle it from here.
Actions
8. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 10, 2014 4:14 PM (in response to clebert.suconic)

Actually TempQueue.clear is already deleting the cache. Maybe you just have lots of temp queues?
Actions
9. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 10, 2014 4:18 PM (in response to clebert.suconic)

are u deleting your temp queues? there are two ways to delete a temp queue... by closing the session... or calling TemporaryQueue.delete.
Actions
10. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jcrisp Apr 10, 2014 4:25 PM (in response to clebert.suconic)

We started out with just closing the session. When we came across this problem I added the below to see if any behavior would change:

finally {
            quietlyCloseMessageConsumer(messageConsumer);
            quietlyCloseProducer(producer);
            quietlyDeleteQueue(replyQueue); // TempQueue
            quietlyCloseSession(session);
            quietlyCloseConnection(connection);
        }

We do not call a TempQueue.clear anywhere in the code.
Actions
11. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 10, 2014 5:32 PM (in response to jcrisp)

what's the definition of quietlyDeleteQueue ? can you post the code?
Actions
12. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jcrisp Apr 10, 2014 6:10 PM (in response to clebert.suconic)

Here is the code:

public static void quietlyDeleteQueue(TemporaryQueue queue) {
        if (null != queue) {
            try {
                queue.delete();
            } catch (JMSException jex) {
                LOG.error("Error deleting TemporaryQueue", jex);
            }
        }
    }
Actions
13. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

clebert.suconic Apr 11, 2014 10:49 AM (in response to jcrisp)

I don't see a leak then... beyond that you are using lots of temp queues maybe? you should look for available patches on HornetQ available for your EAP.
1 of 1 people found this helpful
Actions
14. Re: While load testing on EAP 6.2.0 / 6.2.2 get memory leaks that point to HornetQ messaging use-duplicate-detection setting

jbertram Apr 14, 2014 5:15 PM (in response to clebert.suconic)

The underlying leak of duplicate IDs has been fixed via https://issues.jboss.org/browse/HORNETQ-1342.
Actions

Go to original post