10 Replies Latest reply on Feb 28, 2006 7:40 AM by lauri

    Memory leak and tons of org.jboss.mq.SpyTopic and org.jboss.

    lauri

      Hello people.

      I'm new to JBoss (~month of experience), however i've migrated production installation of project to JBoss from SunApplication server.
      It runs much better, but we're now facing memory leak.
      We're running jboss as application server and some socket daemons as clients for the server.
      Now both (client and server) consumes memory. I've compared "jmap -histo pid" results for client after 12 hours of heavy load.
      It is obivious that amount
      org.jboss.mq.SpyTopic and
      org.jboss.mq.AcknowledgementRequest has grown
      Was (1column is bytes, 2nd - amount of instances, 3d - class name):

      182016 7584 org.jboss.mq.SpyTopic
      181992 7583 org.jboss.mq.AcknowledgementReque

      After 12 hours

      917760 38240 org.jboss.mq.SpyTopic
      917736 38239 org.jboss.mq.AcknowledgementRequest

      The client-server uses JMS communication a lot.
      Client message handler:
      public void onMessage(javax.jms.Message message) {
       try {
       message.acknowledge();
       MapMessage mapMessage = (MapMessage) message;
       //...........actions on message
       } catch (JMSException jmsEx) {
       logger.log(Level.INFO, "Error processing message", jmsEx);
       } catch (ClassCastException cCEx) {
       logger.fine(cCEx.getMessage());
       }
      
       }
      

      Server side (in session and eninty beans) message producer
       private void notifyClientListeners(Integer playerId, String serverId) {
       try {
       TopicSession session = null;
       Topic topic = this.serverTopic; // acquired in ejbCreate from JNDI env.
       TopicConnectionFactory tcf = this.jmsFactory; // acquired in ejbCreate from JNDI env.
       TopicConnection conn = tcf.createTopicConnection(this.jmsUserName, this.jmsUserPassword);
      
       session = conn.createTopicSession(false, TopicSession.AUTO_ACKNOWLEDGE);
       conn.start();
      
       TopicPublisher send = session.createPublisher(topic);
      
       MapMessage mapMessage = session.createMapMessage();
       mapMessage.setString(JMSConstants.JMS_MESSAGE_NAME, JMSConstants.JMS_PLAYER_CONNECT);
       mapMessage.setInt(JMSConstants.JMS_PLAYER_ID, playerId.intValue());
       mapMessage.setString(JMSConstants.JMS_SERVER_ID, serverId);
      
       send.publish(mapMessage);
      
       send.close();
       session.close();
       conn.close();
       } catch (Exception ex) {
       logger.log(Level.WARNING, "Unable to notify servers", ex);
       }
       }
      


      JMS connectivity works using mysql. Here we have a error too. Sometimes following error occures:
      06:25:45,394 WARN [LocalManagedConnectionFactory] Destroying connection that is not valid, due to the following exception: com.mysql.jdbc.Connection@3d7ba3
      com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception:

      ** BEGIN NESTED EXCEPTION **

      java.net.SocketException
      MESSAGE: Connection reset

      STACKTRACE:

      java.net.SocketException: Connection reset
      at java.net.SocketInputStream.read(SocketInputStream.java:113)
      at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113)

      This error occures in many cases, JMS, entiny beans and so on. But we don't have data loss.

        • 1. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
          lauri

          Forgot to place question.
          Does anybody have ideas how to
          1) eliminate memory leaks
          2) connection resets from mysql
          ?
          Thanks for advance.

          • 2. Additional information.
            lauri

            configuration
            Jboss 4.0.3
            JDK 1.5.0_06
            SunOS 5.9 Generic_118559-09 i86pc i386 i86pc
            mysql 4.1

            I've noticed that SELECT count(*) FROM JMS_MESSAGES returns amount of org.jboss.mq.AcknowledgementRequest and org.jboss.mq.SpyTopic objects in
            jmap -histo output.

            What could it mean?

            • 3. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
              genman


              How does the number of topic objects compare to the number actually in use? Open files? Perhaps you have some reasource leak in your program, not in the app server.

              • 4. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
                lauri

                One leak source has been found.
                Producer sessions were started as (note false)

                session = conn.createTopicSession(false, TopicSession.AUTO_ACKNOWLEDGE);

                Consumer sessions started as (note true)
                session = conn.createTopicSession(true, TopicSession.AUTO_ACKNOWLEDGE);

                and onMessage(message) contains
                message.acknowledge();

                I think that since producer was not transactional and was with AUTO_ACKNOWLEDGE flag, each time when clint call to message.acknowledge(); instance of org.jboss.mq.AcknowledgementRequest was created.
                ---
                I've replased true to false in client side and have removed message.acknowledge(). It helped much!

                • 5. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
                  genman


                  Perhaps this is a bug, since the server should be ignoring acknowledgements from the client if they are not necessary. If you can characterize it as such, can you file a JIRA issue?

                  • 6. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
                    hosierdm

                    Why are you explicitly acknowledging your messages in the client when you clearly have the session set to auto acknowledge? That is not necessary. Don't know if it has anything to do with your problem, but just a tip.

                    • 7. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb

                       

                      "hosierdm" wrote:
                      Why are you explicitly acknowledging your messages in the client when you clearly have the session set to auto acknowledge? That is not necessary. Don't know if it has anything to do with your problem, but just a tip.


                      And ignored by JBossMQ
                       if (session.acknowledgeMode == Session.CLIENT_ACKNOWLEDGE)
                       doAcknowledge();
                      


                      It is more likely the transaction session is not getting committed/rolledback so it is just
                      accumulating message acknowledgements.

                      • 8. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
                        lauri

                         

                        "hosierdm" wrote:
                        Why are you explicitly acknowledging your messages in the client when you clearly have the session set to auto acknowledge?

                        That is undoubtly kind of mistake (or more likely mess) in code. Now it fixed.
                        - - -
                        About JIRA.
                        Adrian, should i place it there or better not?

                        • 9. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb

                           

                          "Lauri" wrote:

                          About JIRA.
                          Adrian, should i place it there or better not?



                          Why? See my previous post.
                          JBossMQ ignores the redundant message.acknowledge()
                          This can't be the cause of your problem.


                          • 10. Re: Memory leak and tons of org.jboss.mq.SpyTopic and org.jb
                            lauri

                            For those who will find this topic as answer to mysql "connection reset" search query.
                            The problem has been solved too.

                            Connection have had been really closed by mysql.
                            It was and is set up with WAIT_TIMEOUT less than 1 minute (I guess that admin had tried to minimize connection leaks for php ?)

                            I have changed connection string in -ds.xml file to

                            jdbc:mysql://spinity.com:3306/java?autoReconnectForPools=true&sessionVariables=wait_timeout=28800

                            (don't forget to mark the string as CDATA)
                            note sessionVariables=wait_timeout=28800
                            I'm not sure that autoReconnectForPools is a good idea.
                            Useful links
                            http://dev.mysql.com/doc/refman/5.0/en/cj-configuration-properties.html
                            http://wiki.jboss.org/wiki/Wiki.jsp?page=SetUpAMysqlDatasource (here stated that autoReconnect is not a good idea, but nothing about autoReconnectForPools).