SystemUsage memory limit reached

[adrianwoodhead]

I had an Active MQ 5.2.0 server and kept running into an issue which I described in detail here and never got a response to. I then moved over to using FUSE in the hope that this would solve the issue. I have run FUSE Message Broker 5.3.0.2 for the last week and today the issue came back, albeit with a slightly different stack trace:

WARN  Service                        - Async error occurred: javax.jms.ResourceAllocationException: SystemUsage memory limit reached

javax.jms.ResourceAllocationException: SystemUsage memory limit reached

        at org.apache.activemq.broker.region.Queue.send(Queue.java:407)

        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)

        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)

        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)

        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)

        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)

        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)

        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)

        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)

        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)

        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)

        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)

        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)

        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)

        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)

        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)

        at java.lang.Thread.run(Thread.java:619)

The config, server machine, client load etc. are all the same as described in the original post, the only difference is that I swapped ActiveMQ out and replaced it with FUSE. Hopefully someone on these forums will be able to help. If any further information is required I am happy to do whatever is necessary to provide it.

[socallag]

Hi,

You can change the memory limits on the systemUsage by modifying the activemq.xml configuration file. Please see the following for more details:  http://activemq.apache.org/producer-flow-control.html

Regards,

Seán.

[adrianwoodhead]

Yes, I have read that document (numerous times ).

Our systemusage entry looks like so:

There is plenty of RAM on the machine, to me this looks like a memory leak somewhere in ActiveMQ as it runs fine for a week or ten days and then all of a sudden stops responding with that error. I don't get any errors in the clients, the server just turns into a black hole where all messages vanish into, the only way I can tell something has gone wrong is by looking at the log file. I then restart it and everything is fine until another week or so.

[socallag]

Hi Adrian,

If it is a memory leak is it possible to profile the application and find out where this leak is or send us in a minimal testcase which we can use to reproduce the issue?

Regards,

Seán.

[adrianwoodhead]

If you look in the original post on the ActiveMQ forum I included jstack and jmap output, I was hoping that would give a clue as to the problem. I can do the same against the FUSE server instance next time it goes into that state if you would like.

I'm afraid this is something hard to reproduce via a test case. What I originally did was issue jstack/jmap every few hours for a few days and didn't notice anything suspicious. I then left it for a few more days, and then I got that message. If I do a few thousand requests its all fine, it seems to happen when left for long periods of time with a sustained load of tens to hundreds of messages a second. If you need any more info about how we send and consume messages, architecture, configuration just let me know.