1 2 3 Previous Next 35 Replies Latest reply on Jan 11, 2013 2:49 PM by apparaonali

    Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78

    apparaonali Newbie

      Hi,

        I am using jboss7.1.2, I am observing some meory issues in Jboss7.

       

      Whn I run our application for 5 minutes my hep memory is reaching to 1GB. I run Eclipse memory analyser and it is showing suspectable memory leak as below.

       

      Class Name                                                                                                                                         | Shallow Heap | Retained Heap

      -----------------------------------------------------------------------------------------------------------------------------------------

      java.util.Vector @ 0x19840780                                                                                                               |       48 |   101,857,520
      '- classes org.jboss.modules.ModuleClassLoader @ 0x19840698                                  80 |   101,862,304
         |- <classloader> class org.xnio.LimitedBlockingQueue @ 0x45d48b78                                                        |        8 |        32
         |  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a1e7340                                                                    |       40 |        40
         |  |  |- <Java Local> java.lang.Thread @ 0x26a4ae98  Remoting "is3-64w7-regr6:MANAGEMENT" task-4 Thread   |      112 |     1,008
         |  |  |- <Java Local> java.lang.Thread @ 0x26a4af28  Remoting "is3-64w7-regr6:MANAGEMENT" task-3 Thread|      112 |       976
         |  |  |- <Java Local> java.lang.Thread @ 0x26a4afb8  Remoting "is3-64w7-regr6:MANAGEMENT" task-2 Thread|      112 |       976
         |  |  |- <Java Local> java.lang.Thread @ 0x26a4b048  Remoting "is3-64w7-regr6:MANAGEMENT" task-1 Thread|      112 |       976
         |  |  '- Total: 4 entries                                                                          |          |         
         |  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a1e7158                                                                     |       40 |        40
         |  '- Total: 2 entries                                                                             |          |         
         |- <classloader> class org.xnio.XnioWorker$TaskPool @ 0x45d6bab0                                                         |        0 |         0
         |- classloader java.security.ProtectionDomain @ 0x19f3e720                                                                     |       32 |       232

       

      If I run the application for 20-30 minutes then it is showing

       

      Class Name                                                                                            | Shallow Heap | Retained Heap

      -----------------------------------------------------------------------------------------------------------------------------------------

      java.util.Vector @ 0x1995eb58                                                                         |       48 |   457,670,904
      '- classes org.jboss.modules.ModuleClassLoader @ 0x1995ea70                                           |       80 |   457,676,056
         |- <classloader> class org.xnio.LimitedBlockingQueue @ 0x45e5a988                                  |        8 |        32
         |  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a182d70                                           |       40 |        40
         |  |  |- <Java Local> java.lang.Thread @ 0x26ba2160  Remoting "is3-64w7-regr6:MANAGEMENT" task-4 Thread|      112 |       888
         |  |  |- <Java Local> java.lang.Thread @ 0x26ba21f0  Remoting "is3-64w7-regr6:MANAGEMENT" task-3 Thread|      112 |       856
         |  |  |- <Java Local> java.lang.Thread @ 0x26ba22a0  Remoting "is3-64w7-regr6:MANAGEMENT" task-2 Thread|      112 |       696
         |  |  |- <Java Local> java.lang.Thread @ 0x26ba2330  Remoting "is3-64w7-regr6:MANAGEMENT" task-1 Thread|      112 |       952
         |  |  '- Total: 4 entries                                                                          |          |         
         |  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a0eb798                                           |       40 |        40
         |  '- Total: 2 entries                                                                             |          |         
         |- <classloader> class org.xnio.XnioWorker$TaskPool @ 0x45e5c510                                   |        0 |         0
         |- classloader java.security.ProtectionDomain @ 0x19f55ed0                                         |       32 |       232
         '- Total: 3 entries                                                                                |          |         

      -----------------------------------------------------------------------------------------------------------------------------------------

       

      And here is the object where it is consuming more memory.

       

      Class Name                                                                              | Shallow Heap | Retained Heap

      ---------------------------------------------------------------------------------------------------------------------------

      java.lang.Object[160] @ 0x25e89f58                                                      |      656 |   457,670,856
      |- [137] class org.xnio.ByteBufferSlicePool$QueueThread @ 0x4a617cf0                    |        8 |  457,668,848
      |  |- <classloader> org.jboss.modules.ModuleClassLoader @ 0x1995ea70                    |       80 |   457,676,056
      |  |  |- classes java.util.Vector @ 0x1995eb58                                          |       48 |   457,670,904
      |  |  |  |- elementData java.lang.Object[160] @ 0x25e89f58                              |      656 |   457,670,856
      |  |  |  |  |- [137] class org.xnio.ByteBufferSlicePool$QueueThread @ 0x4a617cf0        |        8 |   457,668,848
      |  |  |  |  |  |- <classloader> org.jboss.modules.ModuleClassLoader @ 0x1995ea70        |       80 |   457,676,056
      |  |  |  |  |  |  |- classes java.util.Vector @ 0x1995eb58                              |       48 |   457,670,904
      |  |  |  |  |  |  |  |- elementData java.lang.Object[160] @ 0x25e89f58                  |      656 |   457,670,856
      |  |  |  |  |  |  |  |  |- [137] class org.xnio.ByteBufferSlicePool$QueueThread @ 0x4a617cf0|        8 |   457,668,848

      ---------------------------------------------------------------------------------------------------------------------------

      Eclipse memory analyser shows the above marked objecrs are suspected memory leaks.

      org.xnio.ByteBufferSlicePool$Queue getting added to Vector and size of the vector is getting increased during application run with time.

       

      Memory Analyser doen't show an erefrences to our application code and it is showing to the above objects and class loader.

       

      I want to understand are there any memory leaks in Jboss7.1.2?

       

      is any one observed similar issues in jboss7.1.2?

       

      I took latest jboss7 from nightly build on Nov 16, 2012 and I observed similar issue there also.

       

      Any suggestions or help please.

       

      Thanks for the help in advance


        • 1. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
          Nicklas Karlsson Master

          can you find any references if you look for packages from your application?

           

          What's the structure, what does it do etc?

          • 2. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
            Tomaz Cerar Master

            Hi,

             

            this analisys is after full gc was forced on JVM?

             

            also what does you application do?

            Are you using lots of remoting?

            or anything that would use remoting (cli,remote deploy, maven deploy plugin,....)

             

             

            --

            tomaz

            • 3. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
              apparaonali Newbie

              Eclipse memory analyser is not showing any packages or classes from my application.

               

              Invoking jboss with "-Xms512M -Xmx1024M -XX:MaxPermSize=256M"

              After deployment java process comsumes around 570MB of memory.

              After running our test it reaches to 1GB.

               

              We are using Jboss security to connect to our EJBs, We defined Security domain and authentication details in standalone.xml.

              After connection successfull we invoke the services provided by ejbs.

               

              Our client connect code have the following steps

              1. Creates an ednpoint

              2. gets connection object from endpint

              3. gets ejb client context and register connection at cleint context

              4. Store thread specific client context

               

              Once connect is successfull then our test will invoke all the services provided by ejb.

              Our ejb provided admin services like create/update/delete user and groups. In my test application, I created 1000 users, 100 groups, editing them and deleting them.

               

              I am not sure whther you need the information I provided above or something else. Please be specific what information you are looking for.

              • 4. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                Tomaz Cerar Master

                Hi,

                 

                you say

                apparaonali wrote:

                 

                Our client connect code have the following steps

                1. Creates an ednpoint

                2. gets connection object from endpint

                3. gets ejb client context and register connection at cleint context

                4. Store thread specific client context

                 

                do you ever clear that thread specific context?

                • 5. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                  apparaonali Newbie

                  Thanks for your replies. Please find my comments for all the queries in above comments.

                   

                  Tomaz Cerar >> this analisys is after full gc was forced on JVM?

                  Appa Rao Nali>> Our GC interval is 30 minutes, This heap dump analysis is after 2hours of test run.

                                           Even after 10 hours also Eclipse memory Analyser is shwoing the same above metioned details.

                   

                  Tomaz Cerar >> or anything that would use remoting (cli,remote deploy, maven deploy plugin,....)

                  Appa Rao Nali>> We are not using any one of cli,remote deploy, maven deploy plugin,

                   


                  Tomaz Cerar >>do you ever clear that thread specific context?

                  Appa Rao Nali>> We are storing thread specific client context at client side and the client is getting closed after the test run.

                                            However we are observing the above suspect memory leaks at Jboss7 Server JVM.

                                           We are closing the connection and closeing the endpoint at end of the test run

                                           Do we need to clear ejb client context?

                   

                   

                   

                   


                  • 6. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                    Nicklas Karlsson Master

                    Could you show some code what you are storing and how/where?

                    • 7. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                      apparaonali Newbie

                      Please find the code I used to connect to jboss server.

                       

                      One more information is: Our EJB read/write data from/to PostgreSQl database. Is there any issues corrently in integration of jboss7 with postgresql?

                       

                       

                      package com.server.conn;

                       

                      import java.io.BufferedReader;

                      import java.io.IOException;

                      import java.io.InputStreamReader;

                      import java.net.URI;

                      import java.net.URISyntaxException;

                      import java.util.concurrent.TimeUnit;

                       

                      import javax.security.auth.callback.Callback;

                      import javax.security.auth.callback.CallbackHandler;

                      import javax.security.auth.callback.NameCallback;

                      import javax.security.auth.callback.PasswordCallback;

                      import javax.security.auth.callback.TextInputCallback;

                      import javax.security.auth.callback.TextOutputCallback;

                      import javax.security.auth.callback.UnsupportedCallbackException;

                      import javax.security.sasl.RealmCallback;

                       

                      import org.jboss.ejb.client.EJBClientContext;

                      import org.jboss.ejb.client.EJBClientTransactionContext;

                      import org.jboss.ejb.client.ThreadLocalContextSelector;

                      import org.jboss.ejb.client.remoting.IoFutureHelper;

                      import org.jboss.remoting3.Connection;

                      import org.jboss.remoting3.Endpoint;

                      import org.jboss.remoting3.Remoting;

                      import org.jboss.remoting3.remote.RemoteConnectionProviderFactory;

                      import org.xnio.IoFuture;

                      import org.xnio.OptionMap;

                      import org.xnio.Options;

                       

                       

                       

                       

                      /**

                      * This is main connector to JBoss server.

                      */

                      public class ServerConnector {

                         

                          /**

                           * End point which will used for connections

                           */

                          private Endpoint endpoint;

                         

                          /**

                           * Connection with JBOSS server

                           */

                          private Connection connection;

                       

                          /**

                           * Returns the end point

                           *

                           * @return Instance of <code>Endpoint</code>

                           */

                          public Endpoint getEndpoint() {

                              return endpoint;

                          }

                         

                          /**

                           * Returns the connection

                           *

                           * @return Instance of <code>Connection</code>

                           */

                          public Connection getConnection() {

                              return connection;

                          }

                         

                          private static ThreadLocalContextSelector<EJBClientContext> threadLocalContextSelector = new ThreadLocalContextSelector<EJBClientContext>(

                                  new ThreadLocal<EJBClientContext>() {

                       

                                      @Override

                                      protected EJBClientContext initialValue() {

                                          return EJBClientContext.create();

                                      }

                                  });

                       

                          public static EJBClientContext getEjbClientContext() {

                              return threadLocalContextSelector.getCurrent();

                          }

                         

                          public static OptionMap getConnectionProviderOptions() {

                              return OptionMap.create(Options.SSL_ENABLED, Boolean.FALSE, Options.SASL_SERVER_AUTH, Boolean.FALSE);

                          }

                       

                          public static OptionMap getConnectionOptions() {

                              return OptionMap.create(Options.SASL_POLICY_NOPLAINTEXT, Boolean.FALSE, Options.SASL_POLICY_NOANONYMOUS, Boolean.FALSE);

                          }

                       

                          public void connect(boolean isIdentityPerThread) {

                              try {

                                  endpoint = Remoting.createEndpoint("ejb-remote-client-endpoint", OptionMap.EMPTY);

                                 

                                  endpoint.addConnectionProvider("remote", new RemoteConnectionProviderFactory(), getConnectionProviderOptions());

                                 

                                  final IoFuture<Connection> futureConnection = endpoint.connect(new URI("remote://remoteHost-name:4447"), getConnectionOptions(),

                                                                                                 new MyCallbackHandler("test", "test".toCharArray()) );

                                  connection = IoFutureHelper.get(futureConnection, 5, TimeUnit.SECONDS);

                       

                                  EJBClientContext ejbClientContext = getEjbClientContext();

                                  ejbClientContext.registerConnection(connection);

                                  EJBClientContext.setConstantContext(ejbClientContext);

                                  if (isIdentityPerThread) {

                                      // this selector will allow to create and maintain different connection per thread

                                      EJBClientContext.setSelector(threadLocalContextSelector);

                                  }

                                 

                                  EJBClientTransactionContext localUserTxContext = EJBClientTransactionContext.createLocal();

                                  EJBClientTransactionContext.setGlobalContext(localUserTxContext);

                              } catch (IOException e) {

                                  //Handle exceptions

                                  e.printStackTrace();

                              } catch (URISyntaxException e) {

                                  //Handle exceptions

                                  e.printStackTrace();

                              }

                          }

                         

                          /**

                           * Close connection - should always be called before finishing application

                           */

                          public void close() {

                              if (connection != null) {

                                  try {

                                      connection.close();

                                  } catch (IOException e) {

                                      //Handle exceptions

                                      e.printStackTrace();

                                  }

                              }

                              if (endpoint != null) {

                                  try {

                                      endpoint.close();

                                  } catch (IOException e) {

                                      //Handle exceptions

                                      e.printStackTrace();

                                  }

                              }

                          }

                         

                          public class MyCallbackHandler implements CallbackHandler {

                       

                              private String username;

                              private char[] password;

                             

                              public MyCallbackHandler(String username, char[] password) {

                                  this.username = username;

                                  this.password = password;

                              }

                             

                              /* (non-Javadoc)

                               * @see javax.security.auth.callback.CallbackHandler#handle(javax.security.auth.callback.Callback[])

                               */

                              @Override

                              public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {

                                  // TODO Auto-generated method stub

                                  for (int i = 0; i < callbacks.length; i++) {

                                      if (callbacks[i] instanceof NameCallback) {

                                          NameCallback nc = (NameCallback) callbacks[i];

                                          nc.setName(username);

                                      }

                                      else if (callbacks[i] instanceof TextOutputCallback) {

                                          TextOutputCallback toc = (TextOutputCallback) callbacks[i];

                                          switch (toc.getMessageType()) {

                                          case TextOutputCallback.INFORMATION:

                                              break;

                                          case TextOutputCallback.ERROR:

                                              break;

                                          case TextOutputCallback.WARNING:

                                              break;

                                          default:

                                              throw new IOException("Unsupported message type: " + toc.getMessageType());

                                          }

                                      }

                                      else if (callbacks[i] instanceof PasswordCallback) {

                                          PasswordCallback pc = (PasswordCallback) callbacks[i];

                                          pc.setPassword(password);

                                      }

                                      else if (callbacks[i] instanceof RealmCallback) {

                                          RealmCallback rcb = (RealmCallback) callbacks[i];

                                          if (rcb.getText() == null)

                                              rcb.setText(rcb.getDefaultText());

                                      }

                                      else if (callbacks[i] instanceof TextInputCallback) {

                                          // Prompt the user for the username

                                          TextInputCallback callback = (TextInputCallback) callbacks[i];

                                          callback.setText((new BufferedReader(new InputStreamReader(System.in))).readLine());

                                      }

                                      else {

                                          throw new UnsupportedCallbackException(callbacks[i], "Unrecognized Callback in MyCallbackHandler.");

                                      }

                                  }

                              }

                          }

                         

                      }

                      • 8. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                        apparaonali Newbie

                        Ther is an issue currently on XA Datsource memory leak in Jboss7 (https://community.jboss.org/thread/205085).

                         

                        We did not define data source connection recovery mentioned in the XA Datsource memory leak discussion.

                        Is the issue I reported has any relation to the above issue?

                        • 9. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                          apparaonali Newbie

                          I looked at existing remoting issues in jboss community and found that there is a memory leak fixed in remoting and the fix is available in latest nightly build.

                          I took the nightly build on Nov 30th and here are my observations.

                          • Now the above reported memory leak got fixed and I am no more observing that in MAT(Memory Analyser Tool).
                          • Now with latest JBoss7 I could see a bit improvement, we are getting the error "java.lang.OutOfMemoryError: Java heap space" after 1 hour and 30 minutes. However earlier with Jboss7.1.2 I was getting the error after 1 Hour.

                           

                          I analysed heap using MAT(Memory Analyser Tool) and here are major suspected memory leaks.

                           

                          None of the suspcets are showing our code, Two suspects are in JMX and one is in Hibernate.I don't understand what to do next and I am looking for some inputs and pointers to work and investigate further on this.


                          Please write your suggestions/comments/pointers.

                           


                          Suspect1: HashMap couting is getting increasing run by run and even after running GC using Jconsole.

                          20,691 instances of "org.xnio.nio.NioXnioWorker", loaded by "org.jboss.modules.ModuleClassLoader @ 0x19b70f18" occupy 585,581,712 (57.51%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

                          Keywords
                          org.xnio.nio.NioXnioWorker
                          java.util.HashMap$Entry[]
                          org.jboss.modules.ModuleClassLoader @ 0x19b70f18

                           

                          Common Path To the Accumulation Point
                          10,00048480,000448
                          \ 10,00016480,000120
                          .\ 10,00016480,00016
                          ..\ 10,00032480,0002,480,336
                          ...\ 10,00040480,0002,479,416
                          ....\ 10,00024480,0002,478,824
                          .....\ 10,00040480,0002,478,800
                          ......\ 10,00080480,0002,478,728
                          .......\ 10,00024480,0002,283,624
                          ........\ 10,00040480,0002,283,008
                          .........\ 10,000131,088480,0002,282,952
                          ..........+ 524240520
                          ..........+ 524240624
                          ..........+ 424192416
                          ..........+ 424192416
                          ..........+ 424192520
                          ..........+ 424192520
                          ..........+ 424192416
                          ..........+ 424192520
                          ..........+ 424192416
                          ..........+ 424192416
                          ..........+ 424192416
                          ..........+ 424192416
                          ..........+ 424192416
                          ..........+ 424192416
                          ..........+ 324144416
                          ..........+ 324144312
                          ..........+ 324144520
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........+ 324144312
                          ..........\
                          • Total: 25 of 8,631 entries; 8,606 more
                          10,000207,144480,000

                           

                           

                          Suspect2: HashMap couting is getting increasing run by run and even after running GC using Jconsole.

                          20,691 instances of "org.jboss.remoting3.EndpointImpl", loaded by "org.jboss.modules.ModuleClassLoader @ 0x19a26dc8" occupy 144,665,264 (14.21%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>

                          Keywords
                          org.jboss.remoting3.EndpointImpl
                          java.util.HashMap$Entry[]
                          org.jboss.modules.ModuleClassLoader @ 0x19a26dc8

                          Common Path To the Accumulation Point
                          10,00048880,000448
                          \ 10,00016880,000120
                          .\ 10,00016880,00016
                          ..\ 10,00032880,0002,480,336
                          ...\ 10,00040880,0002,479,416
                          ....\ 10,00024880,0002,478,824
                          .....\ 10,00040880,0002,478,800
                          ......\ 10,00080880,0002,478,728
                          .......\ 10,00024880,0002,283,624
                          ........\ 10,00040880,0002,283,008
                          .........\ 10,000131,088880,0002,282,952
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 424352416
                          ..........+ 324264312
                          ..........+ 324264416
                          ..........+ 324264520
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........+ 324264312
                          ..........\
                          • Total: 25 of 8,557 entries; 8,532 more
                          10,000205,368880,000

                           

                           

                           

                          Suspect3: org.hibernate.internal.util.collections.BoundedConcurrentHashMap couting is getting increasing run by run and even after running GC using Jconsole.

                          Class Name                                                                                              shallow |   Retained |  %     
                          org.hibernate.internal.SessionFactoryImpl @ 0x1e6a24a0                                        |      144 |95,277,536 |  9.36%
                          |- org.hibernate.engine.query.spi.QueryPlanCache @ 0x1e70ffc8                                 |       24 |89,476,384 |  8.79%
                          |  |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap @ 0x1e7990d0           |       40 |89,468,744 |  8.79%
                          |  |  '- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment[32] @ 0x1e8a52b0|      144 |89,468,704 |  8.79%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986838 |       48 | 2,812,648 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9868f8 |       48 | 2,811,016 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986598 |       48 | 2,809,832 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986958 |       48 | 2,809,480 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9864d8 |       48 | 2,807,016 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986778 |       48 | 2,807,016 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9865f8 |       48 | 2,804,200 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9866b8 |       48 | 2,804,200 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986aa8 |       48 | 2,804,192 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986568 |       48 | 2,801,384 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986628 |       48 | 2,801,384 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9867d8 |       48 | 2,801,384 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986718 |       48 | 2,800,952 |  0.28%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9867a8 |       48 | 2,799,632 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9868c8 |       48 | 2,798,920 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986808 |       48 | 2,798,568 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986a78 |       48 | 2,798,568 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9865c8 |       48 | 2,797,688 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9869b8 |       48 | 2,796,464 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986a48 |       48 | 2,795,752 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986928 |       48 | 2,795,288 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986508 |       48 | 2,794,888 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986748 |       48 | 2,794,192 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986658 |       48 | 2,792,936 |  0.27%
                          |  | |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986538 |       48 | 2,790,064 |  0.27%
                          |  | '- Total: 25 of 76 entries; 51 more                                                  |          |           |      
                          • 10. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                            Scott Marlow Master

                            None of the suspcets are showing our code, Two suspects are in JMX and one is in Hibernate.I don't understand what to do next and I am looking for some inputs and pointers to work and investigate further on this.


                            Please write your suggestions/comments/pointers.

                            For the JMX related suspects (1+2), drill deeper into the org.jboss storage associated with these leaks to get more information about the leaked storage.  MAT will show show the outgoing memory references for any entry in the table that you copy/pasted above.  Use a left mouse click to show the object instance values in the left hand side pane.  Use a right mouse click to show the menu of different options. "incoming" references shows everything that points at the object and "outgoing" references is what you want to follow a few levels deeper than you did.  Paste some unique details that you see (not object instance ids but perhaps some name values would be helpful). 

                             

                            When the application is running, what kinds of requests are hitting the application server?  Are there any JMX client requests coming in as well?

                             

                            Regarding suspect 3, the Hibernate QueryPlanCache is expected to grow (doesn't sound like a leak).  You can set some configuration options if you want to reduce the number of entries stored in the query plan cache.  In your persistence.xml or Hibernate configuration file, set "hibernate.query.plan_cache_max_size" to the number of entries query plan entries that should be cached to improve performance.  The default "query plan max size" is currently 2048 (per persistence unit definition in your persistence.xml or per Hibernate Session Factory instance).  The memory used by the QueryPlanCache should be released at application undeloyment time (which is when the persistence unit/Hibernate session factory is closed). 

                             

                            Scott

                            • 11. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                              apparaonali Newbie

                              Thanks for the detailed reply.

                              I am working on the suggestions. Please find

                               

                              Are there any JMX client requests coming in as well?

                              >> I have information about JMX, I opened JConsole to run Full GC as well as to monitor the heap of JVM.

                              >> Are JMX memory refrences due to JConsole?

                              >> I am running the tests again and this time I will not open JConsole to check whether the JMX references are due to JConsole or not.

                               

                               

                              When the application is running, what kinds of requests are hitting the application server?

                              >> We have an EJB which provides administrative services like create/get/edit/delete operations for users/groups.

                              >> The test creates/gets/edits/deletes 1000 users and 100 groups.

                              • 12. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                                Scott Marlow Master

                                From the information given, I wasn't sure if the leak was related to remote JMX clients or not.  I wouldn't expect an OOM to occur just because one JMX client connected (JConsole). 

                                 

                                Instead of waiting for the OOM to occur, startup AS7 and attach MAT to the process (before starting the test).  Explore memory to see what is going on with the org.jboss.as.jmx.PluggableMBeanServerImpl before starting the test.  Do a histogram and get the count of org.jboss.as.jmx.PluggableMBeanServerImpl.  Then start the test but only do a few iterations and get the number of org.jboss.as.jmx.PluggableMBeanServerImpl instances.

                                 

                                When you have an instance count of 10000 objects, it can be harder to find the cause.

                                 

                                Is the client code that is invoking the EJB local or remote?  I'm not sure how your EJB client code could cause this issue but you might check if your closing the InitialContext in a try/finally clause.  I'm not saying to make that change now but it would be good to let us know if the context is getting closed.

                                 

                                For AS6, I blogged about how we reduced the AS6 memory footprint here.  Adrian changed some internal code to not keep a long debug string value around and that helped.  We also cleaned up some JMX related memory as well.  It might not be easy to find the root cause right away, but I think you are on the right track using MAT.

                                 

                                If you can help reduce the AS7 memory footprint further, perhaps you can blog about that after.  Well, lets wait until you figure it out before celebrating. 

                                 

                                Scott

                                • 13. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                                  apparaonali Newbie

                                  Thanks for your quick reply and suggestions.

                                  Here are my comments for some of your queries.

                                  Is the client code that is invoking the EJB local or remote?

                                  >> Remote EJB

                                   

                                  I did some more investigation and here are my observations. Please lokk incoming references at below table.

                                  • org.xnio.nio.NioXnioWorker objects are geting increased with remote requests
                                  • There were around 20 thousand remote requests client sent and below refernces are also around 20 thousand
                                  • I ran a small test case with around 250 remote requests and observed 250 org.xnio.nio.NioXnioWorker objects in retained heap


                                  Can we configure the number of worker threads at server side? If yes, any pointeres?

                                  Is this a known issue?

                                  Are there any workarounds currently?

                                  Any further suggestions/comments?

                                  Are these two issues similar to this issue?

                                  https://issues.jboss.org/browse/REM3-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

                                  https://issues.jboss.org/browse/REM3-161

                                   

                                  Here are l incoming references ..

                                  Class Name                          | Shallow Heap | Retained Heap
                                  org.xnio.nio.NioXnioWorker @ 0x4427cb10 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x442735b0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4426a0e0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44260c10 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44257740 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x442435e8 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4423a1e8 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44230de8 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x442279e8 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4421e518 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4420a630 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44201180 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441f7cb0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441ee7e0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441e52a0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441d1098 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441c7b78 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441be198 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441b4d98 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441ab998 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44197ac0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4418e5f0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44185120 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4417bc50 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44172758 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4415e8a0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441553d0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4414bf00 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44142a30 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44139560 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44125388 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x4411beb8 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x441129e8 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44109518 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x44100048 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440ec160 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440e2c90 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440d97e0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440d0330 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440c6e60 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440b2d68 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440a9968 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440a04d0 |       48 |    28,304
                                  org.xnio.nio.NioXnioWorker @ 0x440970d0 |       48 |    28,304
                                  Total: 44 of 20,720 entries; 20,676 more|          |         

                                   

                                  Each of the above object is having

                                   

                                  Class Name                                                                                      | Shallow Heap | Retained Heap
                                  org.xnio.nio.WorkerThread @ 0xdc52700  Remoting "management-client" write-1                     |      136 |    14,088
                                  |- [0] org.xnio.nio.WorkerThread[1] @ 0xdc503e8                                                 |       16 |    14,104
                                  |  '- writeWorkers org.xnio.nio.NioXnioWorker @ 0xdc4f138                                       |       48 |    28,304
                                  | |- xnioWorker, executor org.jboss.remoting3.remote.RemoteConnectionProvider @ 0xdc4e3f0   |       72 |       464
                                  | |  |- referent java.lang.ref.Finalizer @ 0xdc41eb8                                        |       32 |   804,376
                                  | |  |  |- next java.lang.ref.Finalizer @ 0xdc41678                                         |       32 |   804,408
                                  | |  |  |  |- next java.lang.ref.Finalizer @ 0xdc41638                                      |       32 |   804,440
                                  | |  |  |  |- prev java.lang.ref.Finalizer @ 0xdc41eb8                                      |       32 |   804,376
                                  | |  |  |  '- Total: 2 entries                                                              |          |         
                                  | |  |  |- prev java.lang.ref.Finalizer @ 0xdc41ed8                                         |       32 |   804,344
                                  | |  |  |  |- next java.lang.ref.Finalizer @ 0xdc41eb8                                      |       32 |   804,376
                                  | |  |  |  |- prev java.lang.ref.Finalizer @ 0xdc41ef8                                      |       32 |   804,312
                                  | |  |  |  '- Total: 2 entries                                                              |          |         
                                  | |  |  '- Total: 2 entries                                                                 |          |         
                                  | |  |- this$0 org.jboss.remoting3.remote.RemoteConnectionProvider$1 @ 0xdc4db90            |       16 |        16
                                  | |  |- this$0 org.jboss.remoting3.remote.RemoteConnectionProvider$ProviderInterface @ 0xdc4f170|       16 |        16
                                  | |  '- Total: 3 entries                                                                    |          |         
                                  | |- this$0 org.xnio.XnioWorker$TaskPool @ 0xdc50350                                        |       80 |       672
                                  | |- worker, executor org.jboss.remoting3.EndpointImpl @ 0xdc503f8                          |       88 |     6,992
                                  | |- this$0 org.xnio.XnioWorker$2 @ 0xdc525f8                                               |       16 |        16
                                  | |- this$0 org.xnio.XnioWorker$1 @ 0xdc52608                                               |       24 |        80
                                  | |- worker org.xnio.nio.WorkerThread @ 0xdc52678  Remoting "management-client" read-1      |      136 |    14,088
                                  | |- worker org.xnio.nio.WorkerThread @ 0xdc52700  Remoting "management-client" write-1     |      136 |    14,088
                                  | |- executor, executor org.jboss.remoting3.LocalConnectionProvider @ 0xdd2c850             |       48 |        56
                                  | '- Total: 8 entries                                                                       |          |         
                                  |- me org.xnio.nio.WorkerThread @ 0xdc52700  Remoting "management-client" write-1               |      136 |    14,088
                                  '- Total: 2 entries                                                                             |          |         

                                  -----------------------------------------------------------------------------------------------------------------------------------

                                  • 14. Re: Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78
                                    Scott Marlow Master

                                    I'm not sure how your EJB client code could cause this issue but you might check if your closing the InitialContext in a try/finally clause.  I'm not saying to make that change now but it would be good to let us know if the context is getting closed.

                                     

                                    From your other details, this issue does sound related to your EJB client code.  Check to see if you forgot to close the InitialContext.

                                    1 2 3 Previous Next