Found a way to dead lock some management threads // Memory leak
nigord Feb 11, 2015 3:16 PMHello everyone,
We are starting the migration path to Wildfly starting from a JBOSS 5 version.
Our stack is the following:
- Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
- Wildfly-8.2.0.final
- CentOS Linux release 7.0.1406 (Core) [lastest patches]
We are moving components by components and have deployed about 20 Oracle datasources.
First step was to adapt our monitoring for those datasources utilisation.
We used twiddle.sh before and have moved to using "The twiddle-standalone project on GitHub" as a mean to limit the amount of changed required in our monitoring strategy.
We also do memory monitoring through jboss-cli.sh and have noticed the following memory utilization pattern.
This is repeatable and occur over a 2 days cycle.
The memory histogram list the following as the major memory consumer:
num #instances #bytes class name
----------------------------------------------
1: 65691 770643968 [B
2: 262778 17226488 [Ljava.lang.Object;
Lookinh deeper into this, I found out what I think is the cause:
Found one Java-level deadlock:
=============================
"Remoting "bouvmbuild05:MANAGEMENT" task-14":
waiting to lock monitor 0x00007f13643b0328 (object 0x00000000ecdd8a00, a java.util.ArrayDeque),
which is held by "XNIO-1 I/O-2"
"XNIO-1 I/O-2":
waiting to lock monitor 0x00007f1360758d18 (object 0x00000000ece452c8, a org.xnio.streams.BufferPipeOutputStream),
which is held by "Remoting "bouvmbuild05:MANAGEMENT" task-10"
"Remoting "bouvmbuild05:MANAGEMENT" task-10":
waiting to lock monitor 0x00007f13643b0328 (object 0x00000000ecdd8a00, a java.util.ArrayDeque),
which is held by "XNIO-1 I/O-2"
Java stack information for the threads listed above:
===================================================
"Remoting "bouvmbuild05:MANAGEMENT" task-14":
at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:421)
- waiting to lock <0x00000000ecdd8a00> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnectionHandler.handleConnectionClose(RemoteConnectionHandler.java:114)
at org.jboss.remoting3.remote.RemoteReadListener$1$1.run(RemoteReadListener.java:56)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"XNIO-1 I/O-2":
at org.jboss.remoting3.remote.OutboundMessage.cancel(OutboundMessage.java:288)
- waiting to lock <0x00000000ece452c8> (a org.xnio.streams.BufferPipeOutputStream)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeMessages(RemoteConnectionChannel.java:560)
at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:542)
at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:359)
at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:423)
- locked <0x00000000ecdd8a00> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnectionHandler.receiveCloseRequest(RemoteConnectionHandler.java:213)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:110)
at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:199)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:113)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)
"Remoting "bouvmbuild05:MANAGEMENT" task-10":
at org.jboss.remoting3.remote.RemoteConnection$RemoteWriteListener.send(RemoteConnection.java:294)
- waiting to lock <0x00000000ecdd8a00> (a java.util.ArrayDeque)
at org.jboss.remoting3.remote.RemoteConnection.send(RemoteConnection.java:122)
at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:154)
at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:122)
at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:110)
at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:139)
- locked <0x00000000ece452c8> (a org.xnio.streams.BufferPipeOutputStream)
at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:131)
at org.jboss.remoting3.remote.OutboundMessage.flush(OutboundMessage.java:277)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
at org.jboss.remotingjmx.DelegatingRemotingConnectorServer.writeVersionHeader(DelegatingRemotingConnectorServer.java:208)
at org.jboss.remotingjmx.DelegatingRemotingConnectorServer.access$200(DelegatingRemotingConnectorServer.java:60)
at org.jboss.remotingjmx.DelegatingRemotingConnectorServer$ChannelOpenListener.channelOpened(DelegatingRemotingConnectorServer.java:288)
at org.jboss.remoting3.spi.SpiUtils$ServiceOpenTask.run(SpiUtils.java:126)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Found 1 deadlock.
Now, I was wondering what to do with this?
Do you think this is me doing something terribly wrong or could I be on to some bug?
I have the heap dump, histogram, thread dump available should those help pointing me to the next step.
Thanks ahead,
Maxime
-
jmap_wildfly.heap.txt.zip 799 bytes
-
jmap_wildfly.histo.txt.zip 69.1 KB