10 Replies Latest reply on Oct 31, 2018 2:59 AM by ranjith525

Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud Apr 27, 2017 9:39 AM

Linux 2.6.32-504.8.1.el6.x86_64

java version "1.8.0_65"

wildfly-10.0.0.Final

Hello,

We have noticed the following thread hang. It seems to cause wildfly to stop handling http requests. Only a restart fixes. We have not been able to reproduce the issue under test.

PID = 62566, CPU = 66.7%
"default I/O-72" #175 prio=5 os_prio=0 tid=0x00007f1da09cb800 nid=0xf466 runnable [0x00007f1d7013f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)

locked <0x00007f248f9f8028> (a sun.nio.ch.Util$2)
locked <0x00007f248f9f8010> (a java.util.Collections$UnmodifiableSet)
locked <0x00007f248f95a0a0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:509)

1. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud Apr 27, 2017 1:01 PM (in response to thammoud)

Actually, killing the thread causes the server to recover. Why would one rouge thread hang the entire http subsystem? Any pointers will be greatly appreciated.
Actions
2. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

ctomc Apr 28, 2017 5:52 AM (in response to thammoud)

Can you try with latest JDK8 update as u65 is quite old nowdays
Actions
3. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud Apr 28, 2017 8:10 AM (in response to ctomc)

Hi Tomaz,

We are planning on doing just that as we saw some JDK bugs in that area.
Actions
4. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud May 3, 2017 4:39 AM (in response to thammoud)

We upgraded to the latest JDK. Same issue applies. xnio hung on epoll.
Actions
5. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud May 3, 2017 5:31 PM (in response to thammoud)

We believe that this is the root cause.

[XNIO-265] Accept thread blocks forever on java.nio.channels.SelectableChannel.register - JBoss Issue Tracker
Actions
6. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

ctomc May 3, 2017 5:44 PM (in response to thammoud)

You also use quite old linux kernel.

Try updating it to latest for EL6, or even better if possible migrate to EL7

running something like
yum -y update yum rpm
yum -y clean all
yum -y update
should get you to latest EL6 release and kernel.
Actions
7. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud May 3, 2017 6:02 PM (in response to ctomc)

Hi Tomaz,

What does the kernel have to do with it? The code clearly has a race condition that is leading to this behavior.
Actions
8. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

ctomc May 4, 2017 5:41 AM (in response to thammoud)

It has everything to do with it.

Java NIO utilizes low level features of the operating system to provide such good performance.
Epoll where you are seeing problem is backed by native code in kernel and is quite possible problem lies there.
epoll - Wikipedia
epoll(7) - Linux manual page
Actions
9. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

thammoud Jul 26, 2017 9:30 AM (in response to ctomc)

I have updated [XNIO-265] Accept thread blocks forever on java.nio.channels.SelectableChannel.register - JBoss Issue Tracker to fully explain the root cause of this.
Actions
10. Re: Thread Stuck at sun.nio.ch.EPollArrayWrapper.epollWait

ranjith525 Oct 31, 2018 2:59 AM (in response to thammoud)

Hi Team,
I have getting the below error in linux and oracle jdk 8 .
"High CPU in sun.nio.ch.EPollArrayWrapper.epollWait"
Actions

Go to original post