Wildfly 10.1.0, Java 8, Solaris sparc not processing http requests
caro82 Mar 14, 2018 11:35 AMWe are running on Wildfly 10.1.0, JDK 1.8.0_131, SunOS 5.11 sparc, XNIO 3.4.0.Final.
20 minutes after startup the application is throwing the following errors.
Resource temporarily unavailable: java.io.IOException: Resource temporarily unavailable
at sun.nio.ch.SolarisEventPort.port_create(Native Method) [rt.jar:1.8.0_131]
at sun.nio.ch.EventPortWrapper.<init>(EventPortWrapper.java:97) [rt.jar:1.8.0_131]
at sun.nio.ch.EventPortSelectorImpl.<init>(EventPortSelectorImpl.java:60) [rt.jar:1.8.0_131]
at sun.nio.ch.EventPortSelectorProvider.openSelector(EventPortSelectorProvider.java:36) [rt.jar:1.8.0_131]
at org.xnio.nio.NioXnio$DefaultSelectorCreator.open(NioXnio.java:257)
at org.xnio.nio.NioXnio.getSelector(NioXnio.java:243)
at org.xnio.nio.SelectorUtils.await(SelectorUtils.java:39)
at org.xnio.nio.NioSocketConduit.awaitWritable(NioSocketConduit.java:233)
at io.undertow.protocols.ssl.SslConduit.awaitWritable(SslConduit.java:461) [undertow-core-1.4.0.Final.jar:1.4.0.Final]
at org.xnio.conduits.AbstractSinkConduit.awaitWritable(AbstractSinkConduit.java:66)
at io.undertow.conduits.AbstractFixedLengthStreamSinkConduit.awaitWritable(AbstractFixedLengthStreamSinkConduit.java:275) [undertow-core-1.4.0.Final.jar:1.4.0.Final]
at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87) [undertow-core-1.4.0.Final.jar:1.4.0.Final]
at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1962) [undertow-core-1.4.0.Final.jar:1.4.0.Final]
at org.xnio.channels.Channels.writeBlocking(Channels.java:154)
at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:201) [undertow-servlet-1.4.0.Final.jar:1.4.0.Final]
[...]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.4.0.Final.jar:1.4.0.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) [undertow-servlet-1.4.0.Final.jar:1.4.0.Final]
After several more minutes the application is unable to process any http or https connection.
We extracted a thread dump and there are around 500 threads with the stack similar to the one below:
"default I/O-4" #136 prio=5 os_prio=64 tid=0x0000000105402000 nid=0x9a runnable [0xffffffff41cff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.SolarisEventPort.port_getn(Native Method)
at sun.nio.ch.EventPortWrapper.poll(EventPortWrapper.java:184)
at sun.nio.ch.EventPortSelectorImpl.doSelect(EventPortSelectorImpl.java:71)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x000000075f7c75a8> (a sun.nio.ch.Util$3)
- locked <0x000000075f7c75b8> (a java.util.Collections$UnmodifiableSet)
- locked <0x000000075f7c7568> (a sun.nio.ch.EventPortSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:515)
We end up in the same situation after a restart.
I would appreciate any hint on how to investigate this.