Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads

[khantariquealam]

Hello All,

We are using wildFly 8.2.1 Final in standalone mode. Under heavy load wildfly unable to handle heavy http load.

Below are the configuration and graph where you can see high http response peak. Also http is timeout(given 60 sec in configuration).

In earlier JBOSS version 5.1, we haven't observed this peak and also not http timeout.

Anyone knows the how to handle heavy http loads in Wildfly.

Below are configuration in our environment.

1. Java: 8_66 version

2. Wildfly: 8.2.1-final

3. http-client: 4.3.6

Server Configuration:

Architecture: x86_64

CPU op-mode(s):        32-bit, 64-bit

Byte Order:            Little Endian

CPU(s): 12

On-line CPU(s) list:   0-11

Thread(s) per core:    1

Core(s) per socket:    2

Socket(s): 6

NUMA node(s):          1

Vendor ID: GenuineIntel

CPU family:            6

Model: 45

Stepping: 7

CPU MHz: 2693.509

BogoMIPS: 5387.01

Hypervisor vendor: VMware

Virtualization type:   full

L1d cache: 32K

L1i cache: 32K

L2 cache: 256K

L3 cache: 20480K

NUMA node0 CPU(s):     0-11

Linux Version:

Memory: 20 GB

Red Hat Enterprise Linux Server release 6.6 (Santiago)

TPS : 300
users/threads : 224

GC Parameters:

  JAVA_OPTS="-Xss256K -Xmx16G -Xms16G -XX:MaxMetaspaceSize=512m -XX:+UseG1GC  -XX:+PrintCodeCache -XX:G1ReservePercent=JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true -Dlog4j2.disable.jmx=true -DisThreadContextMapInheritable=true"

JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"

standalone-full.xml Configuration

<subsystem xmlns="urn:jboss:domain:undertow:1.2">

<buffer-cache name="default"/>

<server name="default-server">

<http-listener name="default" socket-binding="http" />

<https-listener name="default-https" socket-binding="https" security-realm="SSLRealm" verify-client="REQUESTED" enabled-cipher-suites="SSL_RSA_WITH_NULL_MD5,SSL_RSA_WITH_NULL_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_RSA_WITH_DES_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA" enabled-protocols="TLSv1.2" max-buffered-request-size="50000" record-request-start-time="true"/>

<host name="default-host" alias="localhost">

<location name="/" handler="welcome-content"/>

<access-log pattern="%h %l %u %t %r %s %b %D" prefix="localhost_access_log"/>

<filter-ref name="server-header"/>

<filter-ref name="x-powered-by-header"/>

</host>

</server>

<servlet-container name="default">

<jsp-config/>

<websockets/>

</servlet-container>

<handlers>

<file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>

</handlers>

<filters>

<response-header name="server-header" header-name="Server" header-value="WildFly/8"/>

<response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>

</filters>

</subsystem>

<subsystem xmlns="urn:jboss:domain:io:1.1">

<worker name="default" io-threads="25" task-max-threads="400"/>

<buffer-pool name="default" buffer-size="16384" buffers-per-slice="128" direct-buffers="true"/>

</subsystem>

<subsystem xmlns="urn:jboss:domain:ejb3:2.0">

            <session-bean>

                <stateless>

                    <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                </stateless>

                <stateful default-access-timeout="5000" cache-ref="simple" passivation-disabled-cache-ref="simple"/>

                <singleton default-access-timeout="5000"/>

            </session-bean>

            <mdb>

                <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:hornetq-ra.rar}"/>

                <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>

            </mdb>

            <pools>

                <bean-instance-pools>

                    <strict-max-pool name="slsb-strict-max-pool" max-pool-size="250" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                    <strict-max-pool name="mdb-strict-max-pool" max-pool-size="250" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                </bean-instance-pools>

            </pools>

            <caches>

                <cache name="simple"/>

                <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>

            </caches>

            <passivation-stores>

                <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>

            </passivation-stores>

            <async thread-pool-name="default"/>

            <timer-service thread-pool-name="default" default-data-store="default-file-store">

                <data-stores>

                    <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>

                </data-stores>

            </timer-service>

            <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>

            <thread-pools>

                <thread-pool name="default">

                    <max-threads count="100"/>

                    <keepalive-time time="100" unit="milliseconds"/>

                </thread-pool>

            </thread-pools>

            <iiop enable-by-default="false" use-qualified-name="false"/>

            <default-security-domain value="other"/>

            <default-missing-method-permissions-deny-access value="true"/>

        </subsystem>

-Tarique

[jaikiran]

It's going to be extremely hard to guess what could be causing this, especially for an old community edition (latest is 10.0.x WildFly) and without any application that consistently reproduces this.

If you can give it a try against the latest released community edition and provide a reproducible application, someone then might be able to help.

[khantariquealam]

Thanks Jaikiran,

But this is definitely a big issue. We are in stress testing and will go for production.

Already migrating from JBOSS 5 to Wildfly 8 took long time. We can't suddenly go for higher version after releasing stable 8.2.1-Final version.

We found wildfly unable to handle concurrent http request even though undertow claim, it can handle 1 million connection.

I think undertow developer can provide some hints to handle this.

Hope you understand this situation and provide some more inputs to scaling wildfly server.

Regards,

Tarique

[ctomc]

Mohd tarique alam Khan wrote:

We found wildfly unable to handle concurrent http request even though undertow claim, it can handle 1 million connection.

I think undertow developer can provide some hints to handle this.

There ware quite few performance related issues found in 8.x and fixed in later releases, that is where Jaikiran's suggestion to use later version comes from.

Undertow can handle lots of concurrent connections/requests but it usually boils down to application that is running, so problem could not be related to web server at all.

Did you do anything to identify hot spots with tools like jprofiler/yourkit to see where most cpu time is spent?

Is it in web server related code or somewhere in your application?

migrating from jboss 5 to 8 is a big change and there could be lots of other factors that could impact results you are seeing, web server is just one component in this whole stack.

[khantariquealam]

See the new graph...it's too ugly to show the performance of Wildfly. Could you please provide the some workaround to handle this scenario?

[mayerw01]

Did you make sure that the issue is not caused by memory leaks?

[khantariquealam]

We don't have memory leaks and server is running without any JVM related exceptions.

[mayerw01]

Here I was rather refering to the Java memory (mainly the heap) and GC. 'Leak Suspects' do not neccessarily throw Java exceptions. I assume you did also upgrade the Java version (a good overview to the memory health may provide the Eclipse Memory Analyzer or even JConsole).

Could you also provide more details which tool you are using,  which figure you are exactly referring and how this figure changed compared to JBOSS 5.1. What elese did change in your environment?

Is this a special application where the issue appears? When speaking about HTTP I assume you are referring to a Web application (HTML page). Did you also try to localise the root cause via your browser (eg firebug)?

[khantariquealam]

Thanks Wolfgang,

Yes we are using JDK8_66 version. Also we have analyzed the heap dump using Eclipse memory analyzer but not found any GC leak suspects(We will go more deeply after 24 hour run) but we short run we have not found any leak suspects.

In earlier JBOSS 5.1, We were using JDK7. Now in new environment, I have provided all the information on my first post.

We are not generating HTML page. Our application is return XML response.

We are using JMeter through which we send http get or Post request to get XML response. Please see below log where you can see server stop responding and apache http client (version 4.3.6) sends connection time out after 70 sec(we have defined in our configuration). Also you can free pool status of httpclient.

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 4

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 6

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 15

2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 5

2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

2016-05-18 14:21:14,652 INFO  [default task-257][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /proxy/uusd - Time taken for Request, /proxy/uusd/getOnTopFavourites : 26ms.

2016-05-18 14:22:24,841 ERROR [default task-74][be.belgacom.tv.brq.helpers.BrqExceptionHandler] [BRQHbrqh181421991] 150001:Communication failed to the proxy.

2016-05-18 14:22:24,867 INFO  [default task-74][be.belgacom.tv.common.BtaErrorMessageResolver] [BRQHbrqh181421991] Old Error Code :150001 New Error Code: 150001 Message: 0xbe100000:System Error.

2016-05-18 14:22:24,937 INFO  [default task-74][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /broker/bta - Time taken for Request, /broker/bta/getOnTopFavourites : 70425ms.

Regards,

Tarique

[khantariquealam]

Thanks Tomaz,

Please see below log where you can see server stop responding and apache http client (version 4.3.6) sends connection time out after 70 sec(we have defined in our configuration). Also you can free pool status of httpclient.

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 4

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 6

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 15

2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 5

2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

2016-05-18 14:21:14,652 INFO  [default task-257][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /proxy/uusd - Time taken for Request, /proxy/uusd/getOnTopFavourites : 26ms.

2016-05-18 14:22:24,841 ERROR [default task-74][be.belgacom.tv.brq.helpers.BrqExceptionHandler] [BRQHbrqh181421991] 150001:Communication failed to the proxy.

2016-05-18 14:22:24,867 INFO  [default task-74][be.belgacom.tv.common.BtaErrorMessageResolver] [BRQHbrqh181421991] Old Error Code :150001 New Error Code: 150001 Message: 0xbe100000:System Error.

2016-05-18 14:22:24,937 INFO  [default task-74][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /broker/bta - Time taken for Request, /broker/bta/getOnTopFavourites : 70425ms.

Regards,

Tarique

[ctomc]

Hey,

all you have is your client benchmark results.

you would really need to do profiling of the java code on the server.

get yourself profiler like Java Profiler - .NET Profiler - The profilers for Java and .NET professionals

for which you can get trial license. and connect it to running WildFly JVM and do CPU profiling while you are running benchmarks.

this will help identify where in the server / application code there is a bottleneck / hotspot.

--

tomaz

[mayerw01]

Are you using a belgacom proxy server. If yes, did you verify that this issue is not related to belgacom?

Maybe you could also try Wireshark?

[khantariquealam]

Thanks Wolfgang,

We have offshore develepment center for belgacom. We are not using any proxy for our application.

Belgacom application contains lots of proxy applications which communicate each other in intranet.

We checked with network team, there is no network issue as well. Today we are going to do profiling for our java application apart from thread and heap dump.

[khantariquealam]

Hello Tomaz,

We have done our testing but the result are the same. After some time (around after 2-3 minutes) with http load using JMeter, Undertow start giving read time out and stream closed error.

Read Time out Error

java.net.SocketTimeoutException: Read timed out

  at java.net.SocketInputStream.$$YJP$$socketRead0(Native Method) ~[?:1.8.0_66]

  at java.net.SocketInputStream.socketRead0(SocketInputStream.java) ~[?:1.8.0_66]

  at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_66]

  at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_66]

  at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_66]

  at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.3.3.jar!/:4.3.3]

  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) ~[httpclient-4.3.6.jar!/:4.3.6]

  at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) ~[httpclient-4.3.6.jar!/:4.3.6]

  at be.belgacom.tv.http.common.BtaPoolingHttpClientManagerFactory.execute(BtaPoolingHttpClientManagerFactory.java:168) ~[bta-common.jar!/:?]

  at be.belgacom.tv.circuitbreaker.wrappers.BtaCBPoolingHttpClientManagerFactory.execute(BtaCBPoolingHttpClientManagerFactory.java:151) [circuitbreaker.jar:?]

  at be.belgacom.tv.brq.services.common.BrqHttpService.forwardHttpGetRequest(BrqHttpService.java:354) [classes:?]

  at be.belgacom.tv.brq.services.common.BrqHttpService.processHttpRequest(BrqHttpService.java:547) [classes:?]

  at be.belgacom.tv.brq.services.impl.AuthenticationServiceBeanImpl.authenticate(AuthenticationServiceBeanImpl.java:68) [classes:?]

  at be.belgacom.tv.brq.interceptor.AuthenticationInterceptorImpl.preHandle(AuthenticationInterceptorImpl.java:120) [classes:?]

  at org.springframework.web.servlet.HandlerExecutionChain.applyPreHandle(HandlerExecutionChain.java:134) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:954) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

  at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

  at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]

  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]

  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at be.belgacom.tv.http.filters.BtaRequestCompressionFilter.doFilter(BtaRequestCompressionFilter.java:88) [http-compression.jar!/:?]

  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-8.2.1.Final.jar!/:8.2.1.Final]

  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-8.2.1.Final.jar!/:8.2.1.Final]

  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:248) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:77) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:167) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:761) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66]

  at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]

Stream Closed Error

2016-05-24 13:10:37,436 ERROR [default task-341][be.belgacom.tv.brq.helpers.BrqExceptionHandler] [null] java.io.IOException: UT010029: Stream is closed

at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:136)

at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)

at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)

at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)

at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)

at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)

at java.io.Writer.write(Writer.java:157)

at org.springframework.util.StreamUtils.copy(StreamUtils.java:105)

at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:109)

at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:40)

at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:208)

at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:161)

at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:101)

at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:185)

at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:71)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:126)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)

at be.belgacom.tv.http.filters.BtaRequestCompressionFilter.doFilter(BtaRequestCompressionFilter.java:88)

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)

at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)

at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70)

at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261)

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:248)

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:77)

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:167)

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:761)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Also please find the CPU, Memory, Heap graph. We are not having any problem on that side.

Similar kind of error was found in Webservice with wildfly 8.2.

http://stackoverflow.com/questions/29303854/wildfly-8-2-undertow-read-time-out

Could you please provide more hint to handle this in undertow?

P.S. We have tried with read time out attribute as well but result is same i.e.

<http-listener name="default" socket-binding="http" max-buffered-request-size="65536" tcp-keep-alive="true" read-timeout="60000"/>

Regards,

Tarique

[khantariquealam]

Hello Everyone,

We are able to solve this problem and it's good to share information on the forum.

1 . Our server was not tuned to cater the maximum number of processes. To overcome this problem, we add the following lines to the /etc/security/limits.conf file:

jboss soft nproc 8000

jboss hard nproc 12000

Remember we already did configuration to tune maximum file descriptors using below line in same /etc/security/limits.conf file

jboss soft nofile 9999

jboss hard nofile 9999

Now after this configuration our server is able to create more threads

<worker name="default" io-threads="20" task-max-threads="1024"/>

2. Timeouts + OS time|close_waits on unused connections

now the the very next "problem" we noticed under heavy load, was the high number of CLOSE|TIME_WAITS on OS level. Tuning http-listeners was solve this problem.

<http-listener name="default" socket-binding="http" record-request-start-time="true" no-request-timeout="2500 request-parse-timeout="25000" tcp-keep-alive="true" read-timeout="30000"/>

We able to solve this problem after <subsystem xmlns="urn:jboss:domain:undertow:1.2" statistics-enabled="true">

3. Our component are also used for messaging so need to address messaging section as well.

To overcome this problem, we added more element in  <subsystem xmlns="urn:jboss:domain:messaging:2.0"> (BOLD are new)

               <persistence-enabled>true</persistence-enabled>

                <thread-pool-max-size>10</thread-pool-max-size>

                <security-enabled>false</security-enabled>

               <statistics-enabled>true</statistics-enabled>

                <journal-file-size>1024000</journal-file-size>

                <journal-directory relative-to="jboss.server.config.dir" path="messagingjournal"></journal-directory>

                <large-messages-directory relative-to="jboss.server.config.dir" path="messaginglargemessages"></large-messages-directory>

                <bindings-directory relative-to="jboss.server.config.dir" path="messagingbindings"></bindings-directory>

                <paging-directory relative-to="jboss.server.config.dir" path="paging"></paging-directory>

                <journal-max-io>500</journal-max-io>

                <journal-min-files>5</journal-min-files>

                <journal-type>ASYNCIO</journal-type>

Now our server is able to handle heavy load and able to achieve more TPS and thread than JBOSS5.1.

Thanks you guys for your support and time.

Regards,

Tarique