[Remoting 3] Possible bug in "org.jboss.ejb.client.EjbClientContext"

[sunil_dixit]

jboss-ejb-client (Version 1.0.18.Final and above) has one bug (garbage collection related) in "public void unregisterEJBReceiver(final EJBReceiver receiver)" API.

This API uses "ExecutorService" to send close notification to registered listeners (if any) asynchronously.

Now if your InitialContext is eligible for garbage collection then it JVM calls RemoteContext.finalize() method which internally result into calling "org.jboss.ejb.client.EJBReceiverContext.close()" which triggers

"org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver()" .

Before "org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver() " gets executed "org.jboss.ejb.client.EJBClientContext.ejbClientContextTasksExecutorService" state is changed from RUNNING to TERMINATED i.e. ( runState is being set to 3) which result into "java.util.concurrent.RejectedExecutionException".

ThreadPoolExecutor.addIfUnderMaximumPoolSize() API will always return false as " if (poolSize < maximumPoolSize && runState == RUNNING)" prevents to create new thread.

Finally caller APIs gets following exception

java.util.concurrent.RejectedExecutionException

        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)

        at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)

        at java.util.concurrent.AbstractExecutorService.submit(Unknown Source)

        at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:432)

        at org.jboss.ejb.client.EJBReceiverContext.close(EJBReceiverContext.java:59)

        at org.jboss.ejb.client.remoting.ChannelAssociation.notifyBrokenChannel(ChannelAssociation.java:404)

        at org.jboss.ejb.client.remoting.ChannelAssociation.access$100(ChannelAssociation.java:59)

        at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:118)

        at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:110)

        at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:501)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:406)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:277)

        at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:515)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

        at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:390)

        at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequest(RemoteConnectionHandler.java:231)

        at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAction(RemoteConnectionHandler.java:376)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

        at org.jboss.remoting3.ConnectionImpl.closeAction(ConnectionImpl.java:52)

        at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)

        at org.jboss.naming.remote.client.HaRemoteNamingStore.closeAsync(HaRemoteNamingStore.java:385)

        at org.jboss.naming.remote.client.NamingStoreCache.release(NamingStoreCache.java:113)

        at org.jboss.naming.remote.client.NamingStoreCache$1.close(NamingStoreCache.java:98)

        at org.jboss.naming.remote.client.RemoteContext.finalize(RemoteContext.java:199)

        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)

        at java.lang.ref.Finalizer.runFinalizer(Unknown Source)

        at java.lang.ref.Finalizer.access$100(Unknown Source)

        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

          Below code in "org.jboss.ejb.client.EJBClientContext" is added into new versions and causing this problem

                // we *don't* want to send these notification to listeners synchronously since the listeners can be any arbitrary

                // application code and can potential block for a long time. So invoke the listeners asynchronously via our ExecutorService

                final Collection<EJBClientContextListener> listeners = new ArrayList<EJBClientContextListener>(this.ejbClientContextListeners);

                for (final EJBClientContextListener listener : listeners) {

                    this.ejbClientContextTasksExecutorService.submit(new Runnable() {

                        @Override

                        public void run() {

                            try {

                                listener.receiverUnRegistered(receiverContext);

                            } catch (Throwable t) {

                                // log and ignore

                                logger.debug("Exception trying to invoke EJBClientContextListener " + listener + " for EJB client context " + EJBClientContext.this + " on un-registertation of EJBReceiver " + receiver, t);

                            }

                        }

                    });

                }

            }

[jmeyer_community]

Sunil,

I believe I may be seeing this issue as well.  Do we know if a bug has been reported for it?  Also, do we know the impact of this "bug?"  Is it potentially harmful to the application when the error is seen?

Thanks!

-Jason-

[sunil_dixit]

Hi Jason,

I have posted the same thing on red hat support team but they are suggested different approach to prevent this exception. In this approach "Programmer has to make InitialContext  class level with static variable instead of method/block level". But I don't found this appropriate.

So your code should look like

Class EJBLookupUtil

{

private static Context ctx;

public static Object doLookup(String jndiName)

{

if(null == ctx)

{

// steps to initialize InitailContext

ctx = new InitialContex();

}

return ctx.lookup(jndiName);

}

}

To fix the problem I have supplied new implementation of "java.util.concurrent.ThreadPoolExecutor" class where "defaultHandler" is changed from "AbortPolicy " to "DiscardPolicy".

i.e. line

private static final RejectedExecutionHandler defaultHandler =  new AbortPolicy();

is changed with

private static final RejectedExecutionHandler defaultHandler =  new DiscardPolicy();

This new class implementation should be used by JVM and "-Xbootclasspath:/p" can be used to override JVM classes.

I hope this helps.

Thanks,

Sunil Dixit

[henrikdeluxe]

Hi,

got the same thing here on my clients.

It makes no difference if i call close() explicit on my InitialContext or if GarbageCollctor does - the exception is the same.

An Workaround like override JVM-Classes wont be my first choice.

I'm using jboss-ejb-client version 1.0.23.Final-redhat-1  (EAP 6.1.1)

Does anyone know if there is an issue in jiira?

[jaikiran]

Please go ahead and file a bug in EJBCLIENT JIRA project

[henrikdeluxe]

Hi, i did not found the rigth Jira to file AS7 Bugs, i only found one for Wildfly

I will contact my RedHat Support with that subject today - may they will provide a patch.

[wdfink]

I've opened [EJBCLIENT-98] java.util.concurrent.RejectedExecutionException if a remote-naming InitialContext should be closed

[henrikdeluxe]

Hi Wolf-Dieter,

so I tell my anxious customers that thier production systems might not be at risk when I supress that exception in thier error logfile?

That isn't exactly my imagination of an professional customer support :-(

Whats my benefit of paying for subscription, when getting the bugfix as recently as anybody else with the next lgpl release?

[wdfink]

The EJBCLIENT-98 is the community upstream issue for that problem. If you use a community version you might be ignore the ERROR, use a provided ejb-client library which contain the fix, or use github integrate the bugfix and rebuild the server by yourself.

If you pay for a subscription this is a different story and you need to address that in the customer support portal, you might reference this issue and ask for a fix for your special version if needed. That is the benefit if you pay with a sbscription for professional support.

[spatwary04]

I am using wildfly 10 with  jboss-ejb-client-2.1.4. I am getting the same issue. This is happening  when i am creating the initialcontext on method level and closing it in finally block.

2016-11-02 13:45:23,621 SEVERE [com.test.TestBean] (EJB default - 21) Error: Service Locator Access Exception: java.io.IOException: Channel Channel ID caa8971a (outbound) of Remoting connection 5c95caa5 to /hostip:port has been closed

    at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:123) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

    at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:115) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

    at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)

    at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:514)

    at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:419)

    at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:290)

    at org.jboss.remoting3.remote.RemoteConnectionChannel.closeReadsAndWrites(RemoteConnectionChannel.java:274)

    at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:534)

    at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:154)

    at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.disassociate(RemotingConnectionEJBReceiver.java:266) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

    at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:446) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

    at org.jboss.ejb.client.EJBClientContext.close(EJBClientContext.java:1295) [jboss-ejb-client-2.1.4.Final.jar:2.1.4.Final]

    at org.jboss.naming.remote.client.ejb.RemoteNamingStoreEJBClientHandler$RemoteNamingEJBClientContextCloseTask.close(RemoteNamingStoreEJBClientHandler.java:118) [jboss-remote-naming-2.0.4.Final.jar:2.0.4.Final]

    at org.jboss.naming.remote.client.RemoteContext.finalize(RemoteContext.java:299) [jboss-remote-naming-2.0.4.Final.jar:2.0.4.Final]

    at java.lang.System$2.invokeFinalize(System.java:1267) [rt.jar:1.8.0_25]

    at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:98) [rt.jar:1.8.0_25]

    at java.lang.ref.Finalizer.access$100(Finalizer.java:34) [rt.jar:1.8.0_25]

    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:210) [rt.jar:1.8.0_25]