6 Replies Latest reply on Apr 25, 2003 8:33 AM by adrian.brock

    Queue and memory leaks

    andrewdem

      Hi, All
      I have a problem with memory leaks when
      using queue in my application.
      The application works on JBoss 3.2.0 RC4 (jdk 1.4.1_02) and Red Hat Linux 7.3
      and using: Filters, Servlets, Queues, MDB.
      Servlet sends messages to queue which will process by MDB.
      All messages processed by MDB.
      I've detected (with Optimizeit Profiler 5.0) that
      char[] most of all allocates heap memory.
      char[] generates by
      org.apache.log4j.CategoryKey() 26.6% (54874 items)
      org.jboss.mq.server.ClientConsumer() 26.54% (54744 items)

      org.jboss.mq.server.ClientConsumer() called from org.jboss.connection.Start()
      (which called from RequestBroker servlet).
      I've closed queue connections in RequestBroker (see source code follow),
      call gc every 5 min, but JVM allocating memory increase every hour.
      After 10 min from start application server JVM allocates about 120M of memory (heap: 30M/22M),
      but in 3-4 days JVM allocate 707M of memory (heap: 104M/56M).
      Every day system is low-activity about 10 hours.
      Once I set Xmx=128M and in one week JVM has crushed with OutOfMemory error.
      This system is not a big and comparatively not very active,
      but needs amount of memory ?!
      See bellow servlet (RequestBroker) that sends messages in queue
      and fragments of jboss conf. files.
      Can you advice to me anything to solve this problem?
      Thanks in advance.

      ================ RequestBroker (servlet) ====================
      <...>
      public class RequestBroker extends HttpServlet
      {
      public void doPost(
      <...>
      {
      <...> QueueConnection con = null;
      QueueSender sender = null;
      Context ctx = null;
      QueueConnectionFactory qcf = null;
      QueueSession session = null;
      Queue queue = null;
      xerces parser = null;
      <..>
      parser = new xerces();
      Document document = null;
      try
      {
      document = parser.parse(new InputSource(in));
      <...>
      }
      msgObject new_msg = null;
      ObjectMessage obj_msg = null;
      try
      {
      //Setup queue
      ctx = new InitialContext();
      qcf =(QueueConnectionFactory) ctx.lookup("queue1"));
      con = qcf.createQueueConnection();
      session =con.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
      queue = (Queue) ctx.lookup(lp.getProperty("queue.mess_to_cpa"));
      con.start();
      sender = session.createSender(queue);
      log.debug("Queue setup finished");
      // Çäåñü ðàçîáðàííûé xml ñêëàäûâàåòñÿ â î÷åðåäü êàê DOM Object
      new_msg = new msgObject(document);
      obj_msg = session.createObjectMessage(new_msg);
      obj_msg.setStringProperty("åå", rr);
      sender.send(obj_msg);
      } catch (Exception e)
      <...>
      } finally
      {
      try
      {
      con.close();
      con = null;
      session.close();
      session = null;
      sender.close();
      sender = null;
      log.debug("Queue connection closed");
      } catch (Exception ee1)
      {
      log.error("Can not clolse queue connection!");
      } //try
      qcf = null;
      queue = null;
      } //try
      ============= msgObject ==================================

      public class msgObject implements Serializable
      {
      private Document doc;
      public msgObject(Document doc)
      {
      this.doc = doc;
      }

      public msgObject()
      {
      this.doc = null;
      }

      public void setDocument(Document doc)
      {
      this.doc = doc;
      }

      public Document getDocument()
      {
      return this.doc;
      }
      };

      ================ jbossmq-service.xml =================

      <depends optional-attribute-name="Invoker">jboss.mq:service=Invoker
      ConnectionFactory
      XAConnectionFactory
      0



      <depends optional-attribute-name="Invoker">jboss.mq:service=Invoker
      RMIConnectionFactory
      RMIXAConnectionFactory
      60000



      <depends optional-attribute-name="Invoker">jboss.mq:service=Invoker
      java:/ConnectionFactory
      java:/XAConnectionFactory
      9097
      60000
      true


      <..>


      100
      150
      <!--
      <depends optional-attribute-name="CacheStore">jboss.mq:service=CacheStore
      -->
      jboss.mq:service=PersistenceManager

      <...>
      =============== jms-service.xml ==========================

      <!-- The server session pool for Message Driven Beans -->

      <depends optional-attribute-name="XidFactory">jboss:service=XidFactory
      StdJMSPool

      org.jboss.jms.asf.StdServerSessionPoolFactory



      ======= ejb-jar.xml (äëÿ MDB) =======================

      <...>

      <message-driven>
      MDB1
      <ejb-name>MDB1</ejb-name>
      <ejb-class>test.MDB1</ejb-class>
      <transaction-type>Container</transaction-type>


      <acknowledge-mode>Auto-acknowledge</acknowledge-mode>
      <message-driven-destination>
      <destination-type>javax.jms.Queue</destination-type>
      <subscription-durability>NonDurable</subscription-durability>
      </message-driven-destination>

      <..>
      ======= jboss.xml =================================


      <enterprise-beans>
      <message-driven>
      <ejb-name>MDB1</ejb-name>
      <destination-jndi-name>queue/queue1</destination-jndi-name>
      <resource-ref>
      <res-ref-name>ConnectionFactory</res-ref-name>
      <jndi-name>ConnectionFactory</jndi-name>
      </resource-ref>
      <resource-ref>
      <res-ref-name>XAConnectionFactory</res-ref-name>
      <jndi-name>XAConnectionFactory</jndi-name>
      </resource-ref>

      </message-driven>

      </enterprise-beans>


      =======================================================
      --
      Best regards,
      Andrey Demchenko.

        • 1. Re: Queue and memory leaks

          Don't cross post.

          This is fixed in 3.2.0 final and 3.0.7

          Regards,
          Adrian

          • 2. Re: Queue and memory leaks
            andrewdem

            Hi,

            Thanks. Problem is solved.

            --
            Best regards,
            Andrey Demchenko.

            • 3. Re: Queue and memory leaks

              Are you sure this is fixed. I experienced the following with 3.2.0.

              Thanks in Advance,
              jj

              13:45:30,003 ERROR [STDERR] java.lang.OutOfMemoryError
              13:49:02,455 WARN [OILServerILService] Connection failure (1).
              java.net.SocketException: Connection reset
              at java.net.SocketInputStream.read(SocketInputStream.java:168)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
              at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133)
              at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316)
              at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383)
              at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455)
              at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604)
              at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
              at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:205)
              at java.lang.Thread.run(Thread.java:536)
              13:49:02,455 WARN [OILServerILService] Connection failure (1).
              java.net.SocketException: Connection reset
              at java.net.SocketInputStream.read(SocketInputStream.java:168)
              at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
              at java.io.BufferedInputStream.read(BufferedInputStream.java:201)
              at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2133)
              at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2316)
              at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2383)
              at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2455)
              at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2604)
              at java.io.ObjectInputStream.readByte(ObjectInputStream.java:845)
              at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:205)
              at java.lang.Thread.run(Thread.java:536)
              13:51:20,621 WARN [OILServerILService] Connection failure (1).
              java.net.SocketException: Connection reset

              • 4. Re: Queue and memory leaks

                What are you jvm and message cache memory
                settings.

                I have explained the message cache too many
                times now, use search.

                Regards,
                Adrian

                • 5. Re: Queue and memory leaks

                  My settings are out of the box jboss 3.2. I see:

                  MaxMemoryMark=600
                  HighMemoryMark=500

                  on the console.

                  I have given no options to java, so it looks like I have the default for -Xmxn which appears to be 64meg.

                  I am sending particularly large messages. It appears I should increase these numbers.

                  I have searched for MaxMemoryMark, for -Xmx and found little of interest. Could you please suggest a few keys for me that point to your explanations?

                  Thanks,
                  jj

                  • 6. Re: Queue and memory leaks

                    How much memory do you have available for jboss?

                    Assuming you have 256M,
                    something like

                    -Xmx256M

                    MaxMemoryMark=240
                    HighMemoryMark=180

                    What this says is that when the memory usage
                    goes over 180M it will try to push messages onto
                    disk to try keep the memory below 240M

                    You need enough difference in the settings so
                    that it isn't suprised by a large message and
                    other operations inside the server have
                    enough working memory.

                    This will only work if the JMS messages are the
                    memory hog. Once all the messages are on
                    disk, there is nothing else the JMS server can do.

                    There I've explained it again, and probably not
                    for the last time :-)

                    Regards,
                    Adrian