1 2 Previous Next 18 Replies Latest reply on Nov 24, 2016 3:13 PM by ajanz

LargeMessageControllerImpl memory consumption

ajanz Sep 16, 2016 7:14 AM

We use JBoss 7.1.1 with hornetq 2.2.13.

we use hornetq/jms to transfer files from one application to another.

last week we got two server shut downs due to OutOfMemory

when i look in heapdump i see 200 LargeMessageControllerImpl with packets/outStream still consuming memory. this could not be active sessions, because the system is only used by much less then 200 users.

so try to repoduce this. i transferred as 230 MB file on my local system and get a heap dump

i the heapdump there are LargeMessageControllerImpl consuming those 230 MB with outStream.

i tried to free those resources but with no luck.

the onMessage code looks like

BytesMessage bm = (BytesMessage) message;
File outputFile = new File(path....);

FileOutputStream fileOutputStream = new FileOutputStream(outputFile);

     BufferedOutputStream bufferedOutput=null;
     try
     {
      bufferedOutput = new BufferedOutputStream(fileOutputStream, 8192);
      bm.setObjectProperty("JMS_HQ_SaveStream", bufferedOutput);

     }
     finally
     {

      fileOutputStream.close();
      bufferedOutput.close();
      bufferedOutput= null;
      fileOutputStream = null ;
     }

any tips what i could do?

ScreenHunter_2520 Sep. 16 13.13.jpg 188.4 KB

1. Re: LargeMessageControllerImpl memory consumption

jbertram Sep 16, 2016 10:00 AM (in response to ajanz)

The first thing I would do is test this use-case with a recent release of Wildfly (e.g. 10.1). JBoss AS 7.1.1 is over 4 years old now. If you still have the problem please provide a reproducible test-case and we'll look into it.
Actions
2. Re: LargeMessageControllerImpl memory consumption

ajanz Sep 20, 2016 7:11 AM (in response to jbertram)

i switched to WildFly 9. but i got nearly same behaviour.

Once a LargeMessageController is created is seems it's buffer is never cleared. maybe this is for reuse reason, but in my tests i got 170 LargeMessageController in the heapdump.

12 of them occupy 11 MB each. so over 200 MB in sum.

is this behaviour as expected?

or how can these buffer cleared?
Actions
3. Re: LargeMessageControllerImpl memory consumption

ajanz Sep 20, 2016 7:19 AM (in response to ajanz)

by the way. my test is with one user, uploading a 230 mb file three times
Actions
4. Re: LargeMessageControllerImpl memory consumption

jbertram Sep 20, 2016 9:30 AM (in response to ajanz)

As I said in my initial reply, if you still have the issue after you upgrade then please provide a reproducible test-case. Without that I can't really answer your other questions because I don't know exactly what you're doing.
Actions
5. Re: LargeMessageControllerImpl memory consumption

ajanz Sep 28, 2016 5:59 AM (in response to jbertram)
of course you are right, i thought you could have may be a quick tip.

so i started to make a test case.

i thought no problem, a half day an everything is done..but then...

the test case works fine. so tried to see where's the difference.

after a few days....i finally found the difference.

we are using a topic. over this topic we send files in a persistent session.

we have different types of MDBs listening to that topic.

some MDBs will process these messages and some will have nothing to do.

and it seems those who have nothing to do cause the problem.

i made a test case for you . deploy the ear file and the test will start with a count down. you need the topic "TOPIC_PERSISTENT" configured.

ScreenHunter_2525 Sep. 28 11.54.jpg 29.4 KB

LargeMessageTest.zip 57.7 KB

LargeMessageTest-ear.ear 17.1 KB
Actions
6. Re: LargeMessageControllerImpl memory consumption

jbertram Sep 28, 2016 6:35 PM (in response to ajanz)

I ran your test a few times while VisualVM was attached to the JVM and I didn't see any significant accumulation related to large messages.
Actions
7. Re: LargeMessageControllerImpl memory consumption

ajanz Sep 29, 2016 2:47 AM (in response to jbertram)

thank you for doing the tests.

i analyzed the memory with jmap and Memory Analyzer ( MAT)
Actions
8. Re: LargeMessageControllerImpl memory consumption

jbertram Sep 29, 2016 10:11 AM (in response to ajanz)

I took heap dumps and analyzed them with VisualVM which is very similar to MAT (which I've used frequently in the past). Like I said, I didn't see any accumulation related to large messages. Do you have or could you create a more conclusive test to demonstrate the problem?
Actions
9. Re: LargeMessageControllerImpl memory consumption

ajanz Sep 30, 2016 5:10 AM (in response to jbertram)
it seems to me that Visual VM isn't precise enough at that point.

i wrote a second test. the ClientTwo now writes the large data to a NullOutputStream.

I ran both tests with Visual VM.

without writing to NullOutputStream the allocation of byte[] was much higher.

LargeMessageTest-ear.ear 17.6 KB

LargeMessageTest (2).zip 59.0 KB
Actions
10. Re: LargeMessageControllerImpl memory consumption

jbertram Sep 30, 2016 12:09 PM (in response to ajanz)
it seems to me that Visual VM isn't precise enough at that point.
I would be surprised by that. VisualVM can take heap dumps and analyze them is essentially the same way as jmap/MAT. I've used both and never noticed any substantive discrepancies.

I ran your test again and I still don't see any accumulation related to large messages.

Couple of thoughts:
You should provide a more conclusive, automated test. This kind of manual testing is very time-consuming and tedious.
Your Maven project does not 'install' out of the box. I have to add a dependency on the JMS API to your ejb module.
The test itself actually throws exceptions due to transaction time-outs. This may be altering the results if you don't see these same exceptions when you test.
Actions
11. Re: LargeMessageControllerImpl memory consumption

ajanz Oct 10, 2016 8:26 AM (in response to jbertram)

of course Visual VM and jmap/MAT should produce the same results. i meant the difference is between VisualVM Sample Data and full heap dump with jmap and MAT.

when you run the first test MAT reports a memory leak which leads from byte[] to ClientConsumerImpl / LargeMessageControllerImpl. see message from 28.09.2016 05:59 there is a screenshot.

in the case "nothing to do" could it be a problem to write to NullOutputStream?
Actions
12. Re: LargeMessageControllerImpl memory consumption

jbertram Oct 10, 2016 11:02 AM (in response to ajanz)

i meant the difference is between VisualVM Sample Data and full heap dump with jmap and MAT.
I looked at the VisualVM sample data, but I also acquired and analyzed heap dumps as well with VisualVM.

I don't have time to run any more manual tests at this point. If you can create an automated test to reproduce the issue (e.g. run continuously until OOME at which point the JVM can generate a heap dump) then I can investigate further.
Actions
13. Re: LargeMessageControllerImpl memory consumption

ajanz Oct 24, 2016 4:56 AM (in response to jbertram)
hello,

i hope you can reproduce this. now i have one message producer , one client really consuming and four clients which do nothing than listen on topic.

i tested on wildfly 9 with 1 GB Memory. after 25 files sended i got an OutOfMemory

i attached an ear file so you can simply deploy it

LargeMessageTest-ear.ear 20.1 KB

LargeMessageTestMDB.zip 164.4 KB
Actions
14. Re: LargeMessageControllerImpl memory consumption

jbertram Oct 24, 2016 10:32 AM (in response to ajanz)

I just ran this with Wildfly 9 using -Xms64m -Xmx512m and it's still running after 83 files sent and no memory issues that I can see.
Actions

1 2 Previous Next

Go to original post