Remoting client-endpoint Threads not closed if a scoped context is closed

[ghansper]

Hello,

we are using the EJB Client API with scoped contexts as described in https://docs.jboss.org/author/display/AS72/Scoped+EJB+client+contexts from a standalone client against JBoss AS 7.2.

We are now getting problems with our test suites that are creating many server connections and finally fail with "java.lang.OutOfMemoryError: unable to create new native thread".

We are observing that though we are closing the InitialContext and the scoped EjbNamingContext as described in above Wiki, the Threads that are associated with the connections are not terminated. There are a couple of 'Remoting "client-endpoint" read/write/task' Daemon Threads associated with each Connection.

Now, we'll be able to circumvent the problem by reducing the amount of connections per test run, but I wonder if there is a way to terminate those Threads?

Regards,

    Gerret

[ghansper]

Hello again,

I have looked a little further into this and found that it is not just creation of a Context which produces the Threads but also each lookup (of a BeanHome in our case) creates six Threads. In addition, each lookup consumes a file handle on the remote server. Since none of the Threads nor the file handles disappear until the client-VM terminates this looks like a rather severe bug to me. Not only that a client can run out of memory, many clients doing many lookups will eventually also cause the server to go out of file handles. Any ideas? Or am I doing something wrong?

By the way: Using the "non scoped" EJB Client API shows the same behaviour when a connection is created, but not for lookups.

Thanks,

    Gerret

[wdfink]

Did you close the InitialContext or the Context which can be looked up with IC.lookup("ejb:") ?

This is the important difference if using scoped-context

[ghansper]

Yes, I'm closing the context from IC.lookup("ejb:") first and then the InitialContext.

[wdfink]

Sounds like [EJBCLIENT-67] OOME (Threads) if scoped EJB context (EJBCLIENT-34) is used with lookups in a loop

which is already fixed.

Are you sure you run the client with the appropriate jboss-client library?

[ghansper]

Yes, that is the one. Thank you, Wolf-Dieter.

I am using 1.0.16 (as came with JBoss AS 7.2 Final afaik). Can you tell me offhand which is the latest version that is compatible with AS 7.2?

[wdfink]

Each 1.0.x library should be compatible at the moment.

Normally you can use the jboss-client.jar bundled with the server, but that is not possible if you have bugs.

For the community version it is a larger gap here until the next version WildFly 8.0.0.Final, you might use that one if you stay with the community version.

[ghansper]

I have built from 1.0.25.Final and this is working.

(Won't switch to WildFly too soon, though. Switching from AS 4 to AS 7 has been a long journey, we need a break )

Interestingly, doing the same thing (creating connections,looking up beans, closing the contexts in a loop) with the non-scoped variant now fails at the 40th connection with "Too many channels open". That used to work with 1.0.16. I have seen e.g. [AS7-3638] Allow configurations for channels created by the EJB client API - JBoss Issue Tracker, and set

remote.connection.<name>.channel.options.org.jboss.remoting3.RemotingOptions.MAX_OUTBOUND_MESSAGES

but this doesn't seem to have an effect. From the code I can see that org.jboss.ejb.client.remoting.ChannelAssociation is initializing a Semaphore with that property's value, but that semaphore never blocks, even for a value of 10 (my test is doing everything sequentially, no concurrent calls).

The code that throws the Exception is in org.jboss.remoting3.remote.RemoteConnectionHandler where

    private final int maxInboundChannels = 40;

    private final int maxOutboundChannels = 40;

never seem to be overwritten - well, they are final anyway . (Using jboss-remoting-3.2.8.GA as referenced from the pom.xml).

Any ideas?

Thanks,

     Gerret

[wdfink]

How your code look like for such non-scoped variant?

[ghansper]

I have attached a jUnit test. Hope it is clear enough how to adjust it for a different environment.