LargeMessageControllerImpl memory consumption

[ajanz]

We use JBoss 7.1.1 with hornetq 2.2.13.

we use hornetq/jms to transfer files from one application to another.

last week we got two server shut downs due to OutOfMemory

when i look in heapdump i see 200 LargeMessageControllerImpl with packets/outStream still consuming memory. this could not be active sessions, because the system is only used by much less then 200 users.

so try to repoduce this.  i transferred as 230 MB file on my local system and get a heap dump

i the heapdump there are LargeMessageControllerImpl consuming those 230 MB with outStream.

i tried to free those resources but with no luck.

the onMessage code looks like

BytesMessage bm = (BytesMessage) message;
     File outputFile = new File(path....);

      FileOutputStream fileOutputStream = new FileOutputStream(outputFile);

     BufferedOutputStream bufferedOutput=null;
     try
     {
      bufferedOutput = new BufferedOutputStream(fileOutputStream, 8192); 
      bm.setObjectProperty("JMS_HQ_SaveStream", bufferedOutput);
     
     }
     finally
     {
     
      fileOutputStream.close();
      bufferedOutput.close();
      bufferedOutput= null;
      fileOutputStream = null ;
     }

any tips what i could do?
    

[jbertram]

The first thing I would do is test this use-case with a recent release of Wildfly (e.g. 10.1).  JBoss AS 7.1.1 is over 4 years old now.  If you still have the problem please provide a reproducible test-case and we'll look into it.

[ajanz]

i switched to WildFly 9. but i got nearly same behaviour.

Once a LargeMessageController is created is seems it's buffer is never cleared. maybe this is for reuse reason, but in my tests i got 170 LargeMessageController in the heapdump.

12 of them occupy 11 MB each. so over 200 MB in sum.

is this behaviour as expected?

or how can these buffer cleared?

[ajanz]

by the way. my test is with one user, uploading a 230 mb file three times

[jbertram]

As I said in my initial reply, if you still have the issue after you upgrade then please provide a reproducible test-case.  Without that I can't really answer your other questions because I don't know exactly what you're doing.

[ajanz]

of course you are right, i thought you could have may be  a quick tip.

so i started to make a test case.

i thought no problem, a half day an everything is done..but then...

the test case works fine. so tried to see where's the difference.

after a few days....i finally found the difference.

we are using a topic. over this topic we send files in a persistent session.

we have different types of MDBs listening to that topic.

some MDBs will process these messages and some will have nothing to do.

and it seems those who have nothing to do cause the problem.

i made a test case for you . deploy the ear file and the test will start with a count down. you need the topic "TOPIC_PERSISTENT" configured.

[jbertram]

I ran your test a few times while VisualVM was attached to the JVM and I didn't see any significant accumulation related to large messages.

[ajanz]

thank you for doing the tests.

i analyzed the memory with jmap and Memory Analyzer ( MAT)

[jbertram]

I took heap dumps and analyzed them with VisualVM which is very similar to MAT (which I've used frequently in the past).  Like I said, I didn't see any accumulation related to large messages.  Do you have or could you create a more conclusive test to demonstrate the problem?

[ajanz]

it seems to me that Visual VM isn't precise enough at that point.

i wrote a second test. the ClientTwo now writes the large data to a NullOutputStream.

I ran both tests with Visual VM.

without writing to NullOutputStream the allocation of byte[] was much higher.

[jbertram]

it seems to me that Visual VM isn't precise enough at that point.

I would be surprised by that.  VisualVM can take heap dumps and analyze them is essentially the same way as jmap/MAT.  I've used both and never noticed any substantive discrepancies.

I ran your test again and I still don't see any accumulation related to large messages.

Couple of thoughts:

• You should provide a more conclusive, automated test.  This kind of manual testing is very time-consuming and tedious.
• Your Maven project does not 'install' out of the box.  I have to add a dependency on the JMS API to your ejb module.
• The test itself actually throws exceptions due to transaction time-outs.  This may be altering the results if you don't see these same exceptions when you test.

[ajanz]

of course Visual VM and jmap/MAT should produce the same results. i meant the difference is between VisualVM Sample Data and full heap dump  with jmap and MAT.

when you run the first test  MAT reports a memory leak which leads from byte[] to ClientConsumerImpl / LargeMessageControllerImpl. see message from 28.09.2016 05:59   there is a screenshot.

in the case "nothing to do" could it be a problem to write to NullOutputStream?

[jbertram]

i meant the difference is between VisualVM Sample Data and full heap dump with jmap and MAT.

I looked at the VisualVM sample data, but I also acquired and analyzed heap dumps as well with VisualVM.

I don't have time to run any more manual tests at this point.  If you can create an automated test to reproduce the issue (e.g. run continuously until OOME at which point the JVM can generate a heap dump) then I can investigate further.

[ajanz]

hello,

i hope you can reproduce this. now i have one message producer , one client really consuming and four clients which do nothing than listen on topic.

i tested on wildfly 9 with 1 GB Memory.  after 25 files sended i got an OutOfMemory

i attached an ear file so you can simply deploy it

[jbertram]

I just ran this with Wildfly 9 using -Xms64m -Xmx512m and it's still running after 83 files sent and no memory issues that I can see.