4 Replies Latest reply on Jun 16, 2009 6:11 AM by Adrian Woodhead

    SystemUsage memory limit reached

    Adrian Woodhead Newbie

      I had an Active MQ 5.2.0 server and kept running into an issue which I described in detail here and never got a response to. I then moved over to using FUSE in the hope that this would solve the issue. I have run FUSE Message Broker 5.3.0.2 for the last week and today the issue came back, albeit with a slightly different stack trace:

       

      WARN  Service                        - Async error occurred: javax.jms.ResourceAllocationException: SystemUsage memory limit reached

      javax.jms.ResourceAllocationException: SystemUsage memory limit reached

              at org.apache.activemq.broker.region.Queue.send(Queue.java:407)

              at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)

              at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)

              at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)

              at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)

              at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)

              at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)

              at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)

              at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)

              at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)

              at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)

              at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)

              at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)

              at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)

              at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)

              at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)

              at java.lang.Thread.run(Thread.java:619)

       

      The config, server machine, client load etc. are all the same as described in the original post, the only difference is that I swapped ActiveMQ out and replaced it with FUSE. Hopefully someone on these forums will be able to help. If any further information is required I am happy to do whatever is necessary to provide it.

        • 1. Re: SystemUsage memory limit reached
          Sean O'Callaghan Newbie

          Hi,

           

          You can change the memory limits on the systemUsage by modifying the activemq.xml configuration file. Please see the following for more details:  http://activemq.apache.org/producer-flow-control.html

           

          Regards,

           

          Seán.

          • 2. Re: SystemUsage memory limit reached
            Adrian Woodhead Newbie

            Yes, I have read that document (numerous times ).

             

            Our systemusage entry looks like so:

             

                 

             

            There is plenty of RAM on the machine, to me this looks like a memory leak somewhere in ActiveMQ as it runs fine for a week or ten days and then all of a sudden stops responding with that error. I don't get any errors in the clients, the server just turns into a black hole where all messages vanish into, the only way I can tell something has gone wrong is by looking at the log file. I then restart it and everything is fine until another week or so.

            • 3. Re: SystemUsage memory limit reached
              Sean O'Callaghan Newbie

              Hi Adrian,

               

              If it is a memory leak is it possible to profile the application and find out where this leak is or send us in a minimal testcase which we can use to reproduce the issue?

               

              Regards,

               

              Seán.

              • 4. Re: SystemUsage memory limit reached
                Adrian Woodhead Newbie

                If you look in the original post on the ActiveMQ forum I included jstack and jmap output, I was hoping that would give a clue as to the problem. I can do the same against the FUSE server instance next time it goes into that state if you would like.

                 

                I'm afraid this is something hard to reproduce via a test case. What I originally did was issue jstack/jmap every few hours for a few days and didn't notice anything suspicious. I then left it for a few more days, and then I got that message. If I do a few thousand requests its all fine, it seems to happen when left for long periods of time with a sustained load of tens to hundreds of messages a second. If you need any more info about how we send and consume messages, architecture, configuration just let me know.