Jboss7 memory leak - class org.xnio.LimitedBlockingQueue @ 0x45d48b78

[apparaonali]

Hi,

  I am using jboss7.1.2, I am observing some meory issues in Jboss7.

Whn I run our application for 5 minutes my hep memory is reaching to 1GB. I run Eclipse memory analyser and it is showing suspectable memory leak as below.

Class Name

| Shallow Heap | Retained Heap

-----------------------------------------------------------------------------------------------------------------------------------------

java.util.Vector @ 0x19840780	|	48 |   101,857,520
'- classes org.jboss.modules.ModuleClassLoader @ 0x19840698	|	80 |   101,862,304
|- <classloader> class org.xnio.LimitedBlockingQueue @ 0x45d48b78	|	8 |	32
|  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a1e7340	|	40 |	40
|  |  |- <Java Local> java.lang.Thread @ 0x26a4ae98  Remoting "is3-64w7-regr6:MANAGEMENT" task-4 Thread   |	112 |	1,008
|  |  |- <Java Local> java.lang.Thread @ 0x26a4af28  Remoting "is3-64w7-regr6:MANAGEMENT" task-3 Thread	|	112 |	976
|  |  |- <Java Local> java.lang.Thread @ 0x26a4afb8  Remoting "is3-64w7-regr6:MANAGEMENT" task-2 Thread	|	112 |	976
|  |  |- <Java Local> java.lang.Thread @ 0x26a4b048  Remoting "is3-64w7-regr6:MANAGEMENT" task-1 Thread	|	112 |	976
|  |  '- Total: 4 entries	|	|
|  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a1e7158	|	40 |	40
|  '- Total: 2 entries	|	|
|- <classloader> class org.xnio.XnioWorker$TaskPool @ 0x45d6bab0	|	0 |	0
|- classloader java.security.ProtectionDomain @ 0x19f3e720	|	32 |	232

If I run the application for 20-30 minutes then it is showing

Class Name

| Shallow Heap | Retained Heap

-----------------------------------------------------------------------------------------------------------------------------------------

java.util.Vector @ 0x1995eb58	|	48 |   457,670,904
'- classes org.jboss.modules.ModuleClassLoader @ 0x1995ea70	|	80 |   457,676,056
|- <classloader> class org.xnio.LimitedBlockingQueue @ 0x45e5a988	|	8 |	32
|  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a182d70	|	40 |	40
|  |  |- <Java Local> java.lang.Thread @ 0x26ba2160  Remoting "is3-64w7-regr6:MANAGEMENT" task-4 Thread|	112 |	888
|  |  |- <Java Local> java.lang.Thread @ 0x26ba21f0  Remoting "is3-64w7-regr6:MANAGEMENT" task-3 Thread|	112 |	856
|  |  |- <Java Local> java.lang.Thread @ 0x26ba22a0  Remoting "is3-64w7-regr6:MANAGEMENT" task-2 Thread|	112 |	696
|  |  |- <Java Local> java.lang.Thread @ 0x26ba2330  Remoting "is3-64w7-regr6:MANAGEMENT" task-1 Thread|	112 |	952
|  |  '- Total: 4 entries	|	|
|  |- <class> org.xnio.LimitedBlockingQueue @ 0x1a0eb798	|	40 |	40
|  '- Total: 2 entries	|	|
|- <classloader> class org.xnio.XnioWorker$TaskPool @ 0x45e5c510	|	0 |	0
|- classloader java.security.ProtectionDomain @ 0x19f55ed0	|	32 |	232
'- Total: 3 entries	|	|

-----------------------------------------------------------------------------------------------------------------------------------------

And here is the object where it is consuming more memory.

Class Name

| Shallow Heap | Retained Heap

---------------------------------------------------------------------------------------------------------------------------

java.lang.Object[160] @ 0x25e89f58	|	656 |   457,670,856
|- [137] class org.xnio.ByteBufferSlicePool$QueueThread @ 0x4a617cf0	|	8 |  457,668,848
|  |- <classloader> org.jboss.modules.ModuleClassLoader @ 0x1995ea70	|	80 |   457,676,056
|  |  |- classes java.util.Vector @ 0x1995eb58	|	48 |   457,670,904
|  |  |  |- elementData java.lang.Object[160] @ 0x25e89f58	|	656 |   457,670,856
|  |  |  |  |- [137] class org.xnio.ByteBufferSlicePool$QueueThread @ 0x4a617cf0	|	8 |   457,668,848
|  |  |  |  |  |- <classloader> org.jboss.modules.ModuleClassLoader @ 0x1995ea70	|	80 |   457,676,056
|  |  |  |  |  |  |- classes java.util.Vector @ 0x1995eb58	|	48 |   457,670,904
|  |  |  |  |  |  |  |- elementData java.lang.Object[160] @ 0x25e89f58	|	656 |   457,670,856
|  |  |  |  |  |  |  |  |- [137] class org.xnio.ByteBufferSlicePool$QueueThread @ 0x4a617cf0|	8 |   457,668,848

---------------------------------------------------------------------------------------------------------------------------

Eclipse memory analyser shows the above marked objecrs are suspected memory leaks.

org.xnio.ByteBufferSlicePool$Queue getting added to Vector and size of the vector is getting increased during application run with time.

Memory Analyser doen't show an erefrences to our application code and it is showing to the above objects and class loader.

I want to understand are there any memory leaks in Jboss7.1.2?

is any one observed similar issues in jboss7.1.2?

I took latest jboss7 from nightly build on Nov 16, 2012 and I observed similar issue there also.

Any suggestions or help please.

Thanks for the help in advance

[nickarls]

can you find any references if you look for packages from your application?

What's the structure, what does it do etc?

[ctomc]

Hi,

this analisys is after full gc was forced on JVM?

also what does you application do?

Are you using lots of remoting?

or anything that would use remoting (cli,remote deploy, maven deploy plugin,....)

--

tomaz

[apparaonali]

Eclipse memory analyser is not showing any packages or classes from my application.

Invoking jboss with "-Xms512M -Xmx1024M -XX:MaxPermSize=256M"

After deployment java process comsumes around 570MB of memory.

After running our test it reaches to 1GB.

We are using Jboss security to connect to our EJBs, We defined Security domain and authentication details in standalone.xml.

After connection successfull we invoke the services provided by ejbs.

Our client connect code have the following steps

1. Creates an ednpoint

2. gets connection object from endpint

3. gets ejb client context and register connection at cleint context

4. Store thread specific client context

Once connect is successfull then our test will invoke all the services provided by ejb.

Our ejb provided admin services like create/update/delete user and groups. In my test application, I created 1000 users, 100 groups, editing them and deleting them.

I am not sure whther you need the information I provided above or something else. Please be specific what information you are looking for.

[ctomc]

Hi,

you say

apparaonali wrote:

Our client connect code have the following steps

1. Creates an ednpoint

2. gets connection object from endpint

3. gets ejb client context and register connection at cleint context

4. Store thread specific client context

do you ever clear that thread specific context?

[apparaonali]

Thanks for your replies. Please find my comments for all the queries in above comments.

Tomaz Cerar >> this analisys is after full gc was forced on JVM?

Appa Rao Nali>> Our GC interval is 30 minutes, This heap dump analysis is after 2hours of test run.

                         Even after 10 hours also Eclipse memory Analyser is shwoing the same above metioned details.

Tomaz Cerar >> or anything that would use remoting (cli,remote deploy, maven deploy plugin,....)

Appa Rao Nali>> We are not using any one of cli,remote deploy, maven deploy plugin,

Tomaz Cerar >>do you ever clear that thread specific context?

Appa Rao Nali>> We are storing thread specific client context at client side and the client is getting closed after the test run.

                          However we are observing the above suspect memory leaks at Jboss7 Server JVM.

                         We are closing the connection and closeing the endpoint at end of the test run

                         Do we need to clear ejb client context?

[nickarls]

Could you show some code what you are storing and how/where?

[apparaonali]

Please find the code I used to connect to jboss server.

One more information is: Our EJB read/write data from/to PostgreSQl database. Is there any issues corrently in integration of jboss7 with postgresql?

package com.server.conn;

import java.io.BufferedReader;

import java.io.IOException;

import java.io.InputStreamReader;

import java.net.URI;

import java.net.URISyntaxException;

import java.util.concurrent.TimeUnit;

import javax.security.auth.callback.Callback;

import javax.security.auth.callback.CallbackHandler;

import javax.security.auth.callback.NameCallback;

import javax.security.auth.callback.PasswordCallback;

import javax.security.auth.callback.TextInputCallback;

import javax.security.auth.callback.TextOutputCallback;

import javax.security.auth.callback.UnsupportedCallbackException;

import javax.security.sasl.RealmCallback;

import org.jboss.ejb.client.EJBClientContext;

import org.jboss.ejb.client.EJBClientTransactionContext;

import org.jboss.ejb.client.ThreadLocalContextSelector;

import org.jboss.ejb.client.remoting.IoFutureHelper;

import org.jboss.remoting3.Connection;

import org.jboss.remoting3.Endpoint;

import org.jboss.remoting3.Remoting;

import org.jboss.remoting3.remote.RemoteConnectionProviderFactory;

import org.xnio.IoFuture;

import org.xnio.OptionMap;

import org.xnio.Options;

/**

* This is main connector to JBoss server.

*/

public class ServerConnector {

    /**

     * End point which will used for connections

     */

    private Endpoint endpoint;

    /**

     * Connection with JBOSS server

     */

    private Connection connection;

    /**

     * Returns the end point

     *

     * @return Instance of <code>Endpoint</code>

     */

    public Endpoint getEndpoint() {

        return endpoint;

    }

    /**

     * Returns the connection

     *

     * @return Instance of <code>Connection</code>

     */

    public Connection getConnection() {

        return connection;

    }

    private static ThreadLocalContextSelector<EJBClientContext> threadLocalContextSelector = new ThreadLocalContextSelector<EJBClientContext>(

            new ThreadLocal<EJBClientContext>() {

                @Override

                protected EJBClientContext initialValue() {

                    return EJBClientContext.create();

                }

            });

    public static EJBClientContext getEjbClientContext() {

        return threadLocalContextSelector.getCurrent();

    }

    public static OptionMap getConnectionProviderOptions() {

        return OptionMap.create(Options.SSL_ENABLED, Boolean.FALSE, Options.SASL_SERVER_AUTH, Boolean.FALSE);

    }

    public static OptionMap getConnectionOptions() {

        return OptionMap.create(Options.SASL_POLICY_NOPLAINTEXT, Boolean.FALSE, Options.SASL_POLICY_NOANONYMOUS, Boolean.FALSE);

    }

    public void connect(boolean isIdentityPerThread) {

        try {

            endpoint = Remoting.createEndpoint("ejb-remote-client-endpoint", OptionMap.EMPTY);

            endpoint.addConnectionProvider("remote", new RemoteConnectionProviderFactory(), getConnectionProviderOptions());

            final IoFuture<Connection> futureConnection = endpoint.connect(new URI("remote://remoteHost-name:4447"), getConnectionOptions(),

                                                                           new MyCallbackHandler("test", "test".toCharArray()) );

            connection = IoFutureHelper.get(futureConnection, 5, TimeUnit.SECONDS);

            EJBClientContext ejbClientContext = getEjbClientContext();

            ejbClientContext.registerConnection(connection);

            EJBClientContext.setConstantContext(ejbClientContext);

            if (isIdentityPerThread) {

                // this selector will allow to create and maintain different connection per thread

                EJBClientContext.setSelector(threadLocalContextSelector);

            }

            EJBClientTransactionContext localUserTxContext = EJBClientTransactionContext.createLocal();

            EJBClientTransactionContext.setGlobalContext(localUserTxContext);

        } catch (IOException e) {

            //Handle exceptions

            e.printStackTrace();

        } catch (URISyntaxException e) {

            //Handle exceptions

            e.printStackTrace();

        }

    }

    /**

     * Close connection - should always be called before finishing application

     */

    public void close() {

        if (connection != null) {

            try {

                connection.close();

            } catch (IOException e) {

                //Handle exceptions

                e.printStackTrace();

            }

        }

        if (endpoint != null) {

            try {

                endpoint.close();

            } catch (IOException e) {

                //Handle exceptions

                e.printStackTrace();

            }

        }

    }

    public class MyCallbackHandler implements CallbackHandler {

        private String username;

        private char[] password;

        public MyCallbackHandler(String username, char[] password) {

            this.username = username;

            this.password = password;

        }

        /* (non-Javadoc)

         * @see javax.security.auth.callback.CallbackHandler#handle(javax.security.auth.callback.Callback[])

         */

        @Override

        public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException {

            // TODO Auto-generated method stub

            for (int i = 0; i < callbacks.length; i++) {

                if (callbacks[i] instanceof NameCallback) {

                    NameCallback nc = (NameCallback) callbacks[i];

                    nc.setName(username);

                }

                else if (callbacks[i] instanceof TextOutputCallback) {

                    TextOutputCallback toc = (TextOutputCallback) callbacks[i];

                    switch (toc.getMessageType()) {

                    case TextOutputCallback.INFORMATION:

                        break;

                    case TextOutputCallback.ERROR:

                        break;

                    case TextOutputCallback.WARNING:

                        break;

                    default:

                        throw new IOException("Unsupported message type: " + toc.getMessageType());

                    }

                }

                else if (callbacks[i] instanceof PasswordCallback) {

                    PasswordCallback pc = (PasswordCallback) callbacks[i];

                    pc.setPassword(password);

                }

                else if (callbacks[i] instanceof RealmCallback) {

                    RealmCallback rcb = (RealmCallback) callbacks[i];

                    if (rcb.getText() == null)

                        rcb.setText(rcb.getDefaultText());

                }

                else if (callbacks[i] instanceof TextInputCallback) {

                    // Prompt the user for the username

                    TextInputCallback callback = (TextInputCallback) callbacks[i];

                    callback.setText((new BufferedReader(new InputStreamReader(System.in))).readLine());

                }

                else {

                    throw new UnsupportedCallbackException(callbacks[i], "Unrecognized Callback in MyCallbackHandler.");

                }

            }

        }

    }

}

[apparaonali]

Ther is an issue currently on XA Datsource memory leak in Jboss7 (https://community.jboss.org/thread/205085).

We did not define data source connection recovery mentioned in the XA Datsource memory leak discussion.

Is the issue I reported has any relation to the above issue?

[apparaonali]

I looked at existing remoting issues in jboss community and found that there is a memory leak fixed in remoting and the fix is available in latest nightly build.

I took the nightly build on Nov 30th and here are my observations.

• Now the above reported memory leak got fixed and I am no more observing that in MAT(Memory Analyser Tool).
• Now with latest JBoss7 I could see a bit improvement, we are getting the error "java.lang.OutOfMemoryError: Java heap space" after 1 hour and 30 minutes. However earlier with Jboss7.1.2 I was getting the error after 1 Hour.

I analysed heap using MAT(Memory Analyser Tool) and here are major suspected memory leaks.

None of the suspcets are showing our code, Two suspects are in JMX and one is in Hibernate.I don't understand what to do next and I am looking for some inputs and pointers to work and investigate further on this.

Please write your suggestions/comments/pointers.

Suspect1: HashMap couting is getting increasing run by run and even after running GC using Jconsole.

20,691 instances of "org.xnio.nio.NioXnioWorker", loaded by "org.jboss.modules.ModuleClassLoader @ 0x19b70f18" occupy 585,581,712 (57.51%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords
org.xnio.nio.NioXnioWorker
java.util.HashMap$Entry[]
org.jboss.modules.ModuleClassLoader @ 0x19b70f18

Common Path To the Accumulation Point

class java.lang.management.ManagementFactory @ 0x445a60e8 System Class	10,000	48	480,000	448
platformMBeanServer org.jboss.as.jmx.PluggableMBeanServerImpl @ 0x19a45670	10,000	16	480,000	120
rootMBeanServer org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer @ 0x19a5b328	10,000	16	480,000	16
delegate com.sun.jmx.mbeanserver.JmxMBeanServer @ 0x19a80ea8	10,000	32	480,000	2,480,336
mbsInterceptor com.sun.jmx.interceptor.DefaultMBeanServerInterceptor @ 0x19aaa010	10,000	40	480,000	2,479,416
repository com.sun.jmx.mbeanserver.Repository @ 0x19ae6a98	10,000	24	480,000	2,478,824
domainTb java.util.HashMap @ 0x19b1f988	10,000	40	480,000	2,478,800
table java.util.HashMap$Entry[16] @ 0x19b62878	10,000	80	480,000	2,478,728
[3] java.util.HashMap$Entry @ 0x19d6b910	10,000	24	480,000	2,283,624
value java.util.HashMap @ 0x19d6b8a8	10,000	40	480,000	2,283,008
table java.util.HashMap$Entry[32768] @ 0x4114c520	10,000	131,088	480,000	2,282,952
[30810] java.util.HashMap$Entry @ 0x2600c030 »	5	24	240	520
[28403] java.util.HashMap$Entry @ 0x13a69f88 »	5	24	240	624
[1545] java.util.HashMap$Entry @ 0xd20c6b8 »	4	24	192	416
[3780] java.util.HashMap$Entry @ 0x1522c9c0 »	4	24	192	416
[24347] java.util.HashMap$Entry @ 0xc8ee618 »	4	24	192	520
[29673] java.util.HashMap$Entry @ 0x1262e458 »	4	24	192	520
[31193] java.util.HashMap$Entry @ 0xbd79070 »	4	24	192	416
[19750] java.util.HashMap$Entry @ 0x56dc818 »	4	24	192	520
[7146] java.util.HashMap$Entry @ 0x12105d18 »	4	24	192	416
[11244] java.util.HashMap$Entry @ 0x16795348 »	4	24	192	416
[11672] java.util.HashMap$Entry @ 0x1140e498 »	4	24	192	416
[10275] java.util.HashMap$Entry @ 0x12bcac38 »	4	24	192	416
[16224] java.util.HashMap$Entry @ 0x16877668 »	4	24	192	416
[9386] java.util.HashMap$Entry @ 0xfa72618 »	4	24	192	416
[11666] java.util.HashMap$Entry @ 0x16c12ac8 »	3	24	144	416
[31967] java.util.HashMap$Entry @ 0x10f145e8 »	3	24	144	312
[12855] java.util.HashMap$Entry @ 0xce16cf8 »	3	24	144	520
[15256] java.util.HashMap$Entry @ 0x10400218 »	3	24	144	312
[9120] java.util.HashMap$Entry @ 0x9953da8 »	3	24	144	312
[23925] java.util.HashMap$Entry @ 0xf52e0b8 »	3	24	144	312
[7800] java.util.HashMap$Entry @ 0x32570748 »	3	24	144	312
[32320] java.util.HashMap$Entry @ 0x10409698 »	3	24	144	312
[31022] java.util.HashMap$Entry @ 0x41f12e88 »	3	24	144	312
[9083] java.util.HashMap$Entry @ 0x8f48eb0 »	3	24	144	312
[3472] java.util.HashMap$Entry @ 0x314cd100 »	3	24	144	312
Total: 25 of 8,631 entries; 8,606 more	10,000	207,144	480,000

Suspect2: HashMap couting is getting increasing run by run and even after running GC using Jconsole.

20,691 instances of "org.jboss.remoting3.EndpointImpl", loaded by "org.jboss.modules.ModuleClassLoader @ 0x19a26dc8" occupy 144,665,264 (14.21%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>

Keywords
org.jboss.remoting3.EndpointImpl
java.util.HashMap$Entry[]
org.jboss.modules.ModuleClassLoader @ 0x19a26dc8

Common Path To the Accumulation Point

class java.lang.management.ManagementFactory @ 0x445a60e8 System Class	10,000	48	880,000	448
platformMBeanServer org.jboss.as.jmx.PluggableMBeanServerImpl @ 0x19a45670	10,000	16	880,000	120
rootMBeanServer org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer @ 0x19a5b328	10,000	16	880,000	16
delegate com.sun.jmx.mbeanserver.JmxMBeanServer @ 0x19a80ea8	10,000	32	880,000	2,480,336
mbsInterceptor com.sun.jmx.interceptor.DefaultMBeanServerInterceptor @ 0x19aaa010	10,000	40	880,000	2,479,416
repository com.sun.jmx.mbeanserver.Repository @ 0x19ae6a98	10,000	24	880,000	2,478,824
domainTb java.util.HashMap @ 0x19b1f988	10,000	40	880,000	2,478,800
table java.util.HashMap$Entry[16] @ 0x19b62878	10,000	80	880,000	2,478,728
[3] java.util.HashMap$Entry @ 0x19d6b910	10,000	24	880,000	2,283,624
value java.util.HashMap @ 0x19d6b8a8	10,000	40	880,000	2,283,008
table java.util.HashMap$Entry[32768] @ 0x4114c520	10,000	131,088	880,000	2,282,952
[8271] java.util.HashMap$Entry @ 0x114e4308 »	4	24	352	416
[32259] java.util.HashMap$Entry @ 0xd5b7ae8 »	4	24	352	416
[3780] java.util.HashMap$Entry @ 0x1522c9c0 »	4	24	352	416
[32513] java.util.HashMap$Entry @ 0x10bb3918 »	4	24	352	416
[22416] java.util.HashMap$Entry @ 0x11cbcac8 »	4	24	352	416
[32486] java.util.HashMap$Entry @ 0xf23ce58 »	4	24	352	416
[12117] java.util.HashMap$Entry @ 0x32f33768 »	4	24	352	416
[19644] java.util.HashMap$Entry @ 0x6ae8b70 »	4	24	352	416
[17632] java.util.HashMap$Entry @ 0xe225568 »	4	24	352	416
[14862] java.util.HashMap$Entry @ 0x2f87d358 »	4	24	352	416
[18184] java.util.HashMap$Entry @ 0x30a90a48 »	4	24	352	416
[32238] java.util.HashMap$Entry @ 0x10209fd8 »	4	24	352	416
[14516] java.util.HashMap$Entry @ 0xc4a1000 »	3	24	264	312
[23562] java.util.HashMap$Entry @ 0xf486b18 »	3	24	264	416
[12855] java.util.HashMap$Entry @ 0xce16cf8 »	3	24	264	520
[27346] java.util.HashMap$Entry @ 0x2ddf3080 »	3	24	264	312
[28672] java.util.HashMap$Entry @ 0x232db590 »	3	24	264	312
[25972] java.util.HashMap$Entry @ 0x10b7bc58 »	3	24	264	312
[6391] java.util.HashMap$Entry @ 0xfe83ea8 »	3	24	264	312
[25780] java.util.HashMap$Entry @ 0x273bceb0 »	3	24	264	312
[14102] java.util.HashMap$Entry @ 0xe4945d8 »	3	24	264	312
[30854] java.util.HashMap$Entry @ 0xca7e388 »	3	24	264	312
[27961] java.util.HashMap$Entry @ 0x11feeec8 »	3	24	264	312
[210] java.util.HashMap$Entry @ 0x7fa9cc0 »	3	24	264	312
[21088] java.util.HashMap$Entry @ 0x123acbf8 »	3	24	264	312
Total: 25 of 8,557 entries; 8,532 more	10,000	205,368	880,000

Suspect3: org.hibernate.internal.util.collections.BoundedConcurrentHashMap couting is getting increasing run by run and even after running GC using Jconsole.

Class Name

shallow |   Retained |  %

org.hibernate.internal.SessionFactoryImpl @ 0x1e6a24a0	|	144 |	95,277,536 |	9.36%
|- org.hibernate.engine.query.spi.QueryPlanCache @ 0x1e70ffc8	|	24 |	89,476,384 |	8.79%
|  |- org.hibernate.internal.util.collections.BoundedConcurrentHashMap @ 0x1e7990d0	|	40 |	89,468,744 |	8.79%
|  |  '- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment[32] @ 0x1e8a52b0|	144 |	89,468,704 |	8.79%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986838 |	48 |	2,812,648 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9868f8 |	48 |	2,811,016 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986598 |	48 |	2,809,832 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986958 |	48 |	2,809,480 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9864d8 |	48 |	2,807,016 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986778 |	48 |	2,807,016 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9865f8 |	48 |	2,804,200 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9866b8 |	48 |	2,804,200 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986aa8 |	48 |	2,804,192 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986568 |	48 |	2,801,384 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986628 |	48 |	2,801,384 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9867d8 |	48 |	2,801,384 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986718 |	48 |	2,800,952 |	0.28%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9867a8 |	48 |	2,799,632 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9868c8 |	48 |	2,798,920 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986808 |	48 |	2,798,568 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986a78 |	48 |	2,798,568 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9865c8 |	48 |	2,797,688 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e9869b8 |	48 |	2,796,464 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986a48 |	48 |	2,795,752 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986928 |	48 |	2,795,288 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986508 |	48 |	2,794,888 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986748 |	48 |	2,794,192 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986658 |	48 |	2,792,936 |	0.27%
|  |	|- org.hibernate.internal.util.collections.BoundedConcurrentHashMap$Segment @ 0x1e986538 |	48 |	2,790,064 |	0.27%
|  |	'- Total: 25 of 76 entries; 51 more	|	|	|

[smarlow]

None of the suspcets are showing our code, Two suspects are in JMX and one is in Hibernate.I don't understand what to do next and I am looking for some inputs and pointers to work and investigate further on this.

Please write your suggestions/comments/pointers.

For the JMX related suspects (1+2), drill deeper into the org.jboss storage associated with these leaks to get more information about the leaked storage.  MAT will show show the outgoing memory references for any entry in the table that you copy/pasted above.  Use a left mouse click to show the object instance values in the left hand side pane.  Use a right mouse click to show the menu of different options. "incoming" references shows everything that points at the object and "outgoing" references is what you want to follow a few levels deeper than you did.  Paste some unique details that you see (not object instance ids but perhaps some name values would be helpful). 

When the application is running, what kinds of requests are hitting the application server?  Are there any JMX client requests coming in as well?

Regarding suspect 3, the Hibernate QueryPlanCache is expected to grow (doesn't sound like a leak).  You can set some configuration options if you want to reduce the number of entries stored in the query plan cache.  In your persistence.xml or Hibernate configuration file, set "hibernate.query.plan_cache_max_size" to the number of entries query plan entries that should be cached to improve performance.  The default "query plan max size" is currently 2048 (per persistence unit definition in your persistence.xml or per Hibernate Session Factory instance).  The memory used by the QueryPlanCache should be released at application undeloyment time (which is when the persistence unit/Hibernate session factory is closed). 

Scott

[apparaonali]

Thanks for the detailed reply.

I am working on the suggestions. Please find

Are there any JMX client requests coming in as well?

>> I have information about JMX, I opened JConsole to run Full GC as well as to monitor the heap of JVM.

>> Are JMX memory refrences due to JConsole?

>> I am running the tests again and this time I will not open JConsole to check whether the JMX references are due to JConsole or not.

When the application is running, what kinds of requests are hitting the application server?

>> We have an EJB which provides administrative services like create/get/edit/delete operations for users/groups.

>> The test creates/gets/edits/deletes 1000 users and 100 groups.

[smarlow]

From the information given, I wasn't sure if the leak was related to remote JMX clients or not.  I wouldn't expect an OOM to occur just because one JMX client connected (JConsole). 

Instead of waiting for the OOM to occur, startup AS7 and attach MAT to the process (before starting the test).  Explore memory to see what is going on with the org.jboss.as.jmx.PluggableMBeanServerImpl before starting the test.  Do a histogram and get the count of org.jboss.as.jmx.PluggableMBeanServerImpl.  Then start the test but only do a few iterations and get the number of org.jboss.as.jmx.PluggableMBeanServerImpl instances.

When you have an instance count of 10000 objects, it can be harder to find the cause.

Is the client code that is invoking the EJB local or remote?  I'm not sure how your EJB client code could cause this issue but you might check if your closing the InitialContext in a try/finally clause.  I'm not saying to make that change now but it would be good to let us know if the context is getting closed.

For AS6, I blogged about how we reduced the AS6 memory footprint here.  Adrian changed some internal code to not keep a long debug string value around and that helped.  We also cleaned up some JMX related memory as well.  It might not be easy to find the root cause right away, but I think you are on the right track using MAT.

If you can help reduce the AS7 memory footprint further, perhaps you can blog about that after.  Well, lets wait until you figure it out before celebrating. 

Scott

[apparaonali]

Thanks for your quick reply and suggestions.

Here are my comments for some of your queries.

Is the client code that is invoking the EJB local or remote?

>> Remote EJB

I did some more investigation and here are my observations. Please lokk incoming references at below table.

• org.xnio.nio.NioXnioWorker objects are geting increased with remote requests
• There were around 20 thousand remote requests client sent and below refernces are also around 20 thousand
• I ran a small test case with around 250 remote requests and observed 250 org.xnio.nio.NioXnioWorker objects in retained heap

Can we configure the number of worker threads at server side? If yes, any pointeres?

Is this a known issue?

Are there any workarounds currently?

Any further suggestions/comments?

Are these two issues similar to this issue?

https://issues.jboss.org/browse/REM3-161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

https://issues.jboss.org/browse/REM3-161

Here are l incoming references ..

Class Name

| Shallow Heap | Retained Heap

org.xnio.nio.NioXnioWorker @ 0x4427cb10 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x442735b0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4426a0e0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44260c10 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44257740 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x442435e8 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4423a1e8 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44230de8 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x442279e8 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4421e518 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4420a630 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44201180 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441f7cb0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441ee7e0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441e52a0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441d1098 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441c7b78 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441be198 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441b4d98 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441ab998 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44197ac0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4418e5f0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44185120 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4417bc50 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44172758 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4415e8a0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441553d0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4414bf00 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44142a30 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44139560 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44125388 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x4411beb8 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x441129e8 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44109518 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x44100048 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440ec160 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440e2c90 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440d97e0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440d0330 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440c6e60 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440b2d68 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440a9968 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440a04d0 |	48 |	28,304
org.xnio.nio.NioXnioWorker @ 0x440970d0 |	48 |	28,304
Total: 44 of 20,720 entries; 20,676 more|	|

Each of the above object is having

Class Name

| Shallow Heap | Retained Heap

org.xnio.nio.WorkerThread @ 0xdc52700  Remoting "management-client" write-1	|	136 |	14,088
|- [0] org.xnio.nio.WorkerThread[1] @ 0xdc503e8	|	16 |	14,104
|  '- writeWorkers org.xnio.nio.NioXnioWorker @ 0xdc4f138	|	48 |	28,304
|	|- xnioWorker, executor org.jboss.remoting3.remote.RemoteConnectionProvider @ 0xdc4e3f0	|	72 |	464
|	|  |- referent java.lang.ref.Finalizer @ 0xdc41eb8	|	32 |	804,376
|	|  |  |- next java.lang.ref.Finalizer @ 0xdc41678	|	32 |	804,408
|	|  |  |  |- next java.lang.ref.Finalizer @ 0xdc41638	|	32 |	804,440
|	|  |  |  |- prev java.lang.ref.Finalizer @ 0xdc41eb8	|	32 |	804,376
|	|  |  |  '- Total: 2 entries	|	|
|	|  |  |- prev java.lang.ref.Finalizer @ 0xdc41ed8	|	32 |	804,344
|	|  |  |  |- next java.lang.ref.Finalizer @ 0xdc41eb8	|	32 |	804,376
|	|  |  |  |- prev java.lang.ref.Finalizer @ 0xdc41ef8	|	32 |	804,312
|	|  |  |  '- Total: 2 entries	|	|
|	|  |  '- Total: 2 entries	|	|
|	|  |- this$0 org.jboss.remoting3.remote.RemoteConnectionProvider$1 @ 0xdc4db90	|	16 |	16
|	|  |- this$0 org.jboss.remoting3.remote.RemoteConnectionProvider$ProviderInterface @ 0xdc4f170|	16 |	16
|	|  '- Total: 3 entries	|	|
|	|- this$0 org.xnio.XnioWorker$TaskPool @ 0xdc50350	|	80 |	672
|	|- worker, executor org.jboss.remoting3.EndpointImpl @ 0xdc503f8	|	88 |	6,992
|	|- this$0 org.xnio.XnioWorker$2 @ 0xdc525f8	|	16 |	16
|	|- this$0 org.xnio.XnioWorker$1 @ 0xdc52608	|	24 |	80
|	|- worker org.xnio.nio.WorkerThread @ 0xdc52678  Remoting "management-client" read-1	|	136 |	14,088
|	|- worker org.xnio.nio.WorkerThread @ 0xdc52700  Remoting "management-client" write-1	|	136 |	14,088
|	|- executor, executor org.jboss.remoting3.LocalConnectionProvider @ 0xdd2c850	|	48 |	56
|	'- Total: 8 entries	|	|
|- me org.xnio.nio.WorkerThread @ 0xdc52700  Remoting "management-client" write-1	|	136 |	14,088
'- Total: 2 entries	|	|

-----------------------------------------------------------------------------------------------------------------------------------

[smarlow]

I'm not sure how your EJB client code could cause this issue but you might check if your closing the InitialContext in a try/finally clause.  I'm not saying to make that change now but it would be good to let us know if the context is getting closed.

From your other details, this issue does sound related to your EJB client code.  Check to see if you forgot to close the InitialContext.