DeadLock: WildFly 10.1.0 EJB Remoting Client
Hi,
Occasionally, our Remote EJB client to our WildFly 10.1.0 Server instance gets deadlocked as shown in the thread dump excerpt below. Can you please let us know whether this is a known issue? If yes, is there a workaround?
Thanks!
Environment:
WildFly 10.1.0 Final
Windows Server 2012 R2
Java 1.8.0_121
Thread Dump Excerpt:
Found one Java-level deadlock:
=============================
"ATM-Data source manager synchronizer":
waiting to lock monitor 0x00000000156bbfc8 (object 0x00000000ec790648, a java.util.ArrayDeque),
which is held by "Remoting "config-based-ejb-client-endpoint" I/O-1"
"Remoting "config-based-ejb-client-endpoint" I/O-1":
waiting to lock monitor 0x00000000156bc1d8 (object 0x00000000f318e250, a org.xnio.streams.BufferPipeOutputStream),
which is held by "ATM-Data source manager synchronizer"
Java stack information for the threads listed above:
===================================================
"ATM-Data source manager synchronizer":
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:300)
- waiting to lock <0x00000000ec790648> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:123)
at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:154)
at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:122)
at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:110)
at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:139)
- locked <0x00000000f318e250> (a org.xnio.streams.BufferPipeOutputStream)
at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:131)
at org.jboss.remoting3.remote.OutboundMessage.flush(OutboundMessage.java:277)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
at org.jboss.marshalling.SimpleDataOutput.flush(SimpleDataOutput.java:339)
at org.jboss.marshalling.SimpleDataOutput.finish(SimpleDataOutput.java:378)
at org.jboss.marshalling.AbstractMarshaller.finish(AbstractMarshaller.java:126)
at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocationInternal(RemotingConnectionEJBReceiver.java:386)
at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.access$100(RemotingConnectionEJBReceiver.java:77)
at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver$2.run(RemotingConnectionEJBReceiver.java:282)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:279)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:184)
at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocation(EJBObjectInterceptor.java:58)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocation(EJBHomeInterceptor.java:83)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:42)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:138)
at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
at com.sun.proxy.$Proxy20.getDataSourcesToRun(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at <<SCRUBBED>>.RemoteAppSessionFactory$ProxyInvocationHandler.invoke(RemoteAppSessionFactory.java:302)
at com.sun.proxy.$Proxy20.getDataSourcesToRun(Unknown Source)
at <<SCRUBBED>>.synchronizeDataSourcesWithServer(DataSourceManager.java:155)
- locked <0x00000000ebfaa760> (a com.onaro.sanscreen.acquisition.framework.mgmt.DataSourceManager)
at <<SCRUBBED>>.run(DataSourceManager.java:763)
at <<SCRUBBED>>.AsynchronousTaskManager$RunnableWrapper.run(AsynchronousTaskManager.java:332)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"Remoting "config-based-ejb-client-endpoint" I/O-1":
at org.jboss.remoting3.remote.OutboundMessage.cancel(OutboundMessage.java:289)
- waiting to lock <0x00000000f318e250> (a org.xnio.streams.BufferPipeOutputStream)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeMessages(RemoteConnectionChannel.java:560)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:542)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:429)
at org.jboss.remoting3.remote.RemoteConnectionHandler.sendCloseRequest(RemoteConnectionHandler.java:233)
at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:113)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:82)
- locked <0x00000000ec790648> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:199)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:113)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)
Found 1 deadlock.