14 Replies Latest reply on Jun 9, 2016 4:33 AM by Mohd tarique alam Khan

    Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads

    Mohd tarique alam Khan Newbie

      Hello All,

       

      We are using wildFly 8.2.1 Final in standalone mode. Under heavy load wildfly unable to handle heavy http load.

      Below are the configuration and graph where you can see high http response peak. Also http is timeout(given 60 sec in configuration).


      In earlier JBOSS version 5.1, we haven't observed this peak and also not http timeout.

      Anyone knows the how to handle heavy http loads in Wildfly.spikeonheavyhttp.jpg


      Below are configuration in our environment.


      1. Java: 8_66 version

      2. Wildfly: 8.2.1-final

      3. http-client: 4.3.6

      Server Configuration:

       

      Architecture: x86_64

      CPU op-mode(s):        32-bit, 64-bit

      Byte Order:            Little Endian

      CPU(s): 12

      On-line CPU(s) list:   0-11

      Thread(s) per core:    1

      Core(s) per socket:    2

      Socket(s): 6

      NUMA node(s):          1

      Vendor ID: GenuineIntel

      CPU family:            6

      Model: 45

      Stepping: 7

      CPU MHz: 2693.509

      BogoMIPS: 5387.01

      Hypervisor vendor: VMware

      Virtualization type:   full

      L1d cache: 32K

      L1i cache: 32K

      L2 cache: 256K

      L3 cache: 20480K

      NUMA node0 CPU(s):     0-11

       

      Linux Version:

      Memory: 20 GB

      Red Hat Enterprise Linux Server release 6.6 (Santiago)

       

      TPS : 300
      users/threads : 224


      GC Parameters:

       

        JAVA_OPTS="-Xss256K -Xmx16G -Xms16G -XX:MaxMetaspaceSize=512m -XX:+UseG1GC  -XX:+PrintCodeCache -XX:G1ReservePercent=JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true -Dlog4j2.disable.jmx=true -DisThreadContextMapInheritable=true"

      JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"

       

      standalone-full.xml Configuration

       

      <subsystem xmlns="urn:jboss:domain:undertow:1.2">

      <buffer-cache name="default"/>

      <server name="default-server">

      <http-listener name="default" socket-binding="http" />

      <https-listener name="default-https" socket-binding="https" security-realm="SSLRealm" verify-client="REQUESTED" enabled-cipher-suites="SSL_RSA_WITH_NULL_MD5,SSL_RSA_WITH_NULL_SHA,SSL_RSA_EXPORT_WITH_RC4_40_MD5,SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA,SSL_RSA_EXPORT_WITH_DES40_CBC_SHA,SSL_RSA_WITH_DES_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA" enabled-protocols="TLSv1.2" max-buffered-request-size="50000" record-request-start-time="true"/>

      <host name="default-host" alias="localhost">

      <location name="/" handler="welcome-content"/>

      <access-log pattern="%h %l %u %t %r %s %b %D" prefix="localhost_access_log"/>

      <filter-ref name="server-header"/>

      <filter-ref name="x-powered-by-header"/>

      </host>

      </server>

      <servlet-container name="default">

      <jsp-config/>

      <websockets/>

      </servlet-container>

      <handlers>

      <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>

      </handlers>

      <filters>

      <response-header name="server-header" header-name="Server" header-value="WildFly/8"/>

      <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>

      </filters>

      </subsystem>

       

      <subsystem xmlns="urn:jboss:domain:io:1.1">

      <worker name="default" io-threads="25" task-max-threads="400"/>

      <buffer-pool name="default" buffer-size="16384" buffers-per-slice="128" direct-buffers="true"/>

      </subsystem>

      <subsystem xmlns="urn:jboss:domain:ejb3:2.0">

                  <session-bean>

                      <stateless>

                          <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>

                      </stateless>

                      <stateful default-access-timeout="5000" cache-ref="simple" passivation-disabled-cache-ref="simple"/>

                      <singleton default-access-timeout="5000"/>

                  </session-bean>

                  <mdb>

                      <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:hornetq-ra.rar}"/>

                      <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>

                  </mdb>

                  <pools>

                      <bean-instance-pools>

                          <strict-max-pool name="slsb-strict-max-pool" max-pool-size="250" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                          <strict-max-pool name="mdb-strict-max-pool" max-pool-size="250" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>

                      </bean-instance-pools>

                  </pools>

                  <caches>

                      <cache name="simple"/>

                      <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>

                  </caches>

                  <passivation-stores>

                      <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>

                  </passivation-stores>

                  <async thread-pool-name="default"/>

                  <timer-service thread-pool-name="default" default-data-store="default-file-store">

                      <data-stores>

                          <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>

                      </data-stores>

                  </timer-service>

                  <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>

                  <thread-pools>

                      <thread-pool name="default">

                          <max-threads count="100"/>

                          <keepalive-time time="100" unit="milliseconds"/>

                      </thread-pool>

                  </thread-pools>

                  <iiop enable-by-default="false" use-qualified-name="false"/>

                  <default-security-domain value="other"/>

                  <default-missing-method-permissions-deny-access value="true"/>

              </subsystem>

       

      -Tarique

        • 1. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
          jaikiran pai Master

          It's going to be extremely hard to guess what could be causing this, especially for an old community edition (latest is 10.0.x WildFly) and without any application that consistently reproduces this.

           

          If you can give it a try against the latest released community edition and provide a reproducible application, someone then might be able to help.

          • 2. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
            Mohd tarique alam Khan Newbie

            Thanks Jaikiran,

            But this is definitely a big issue. We are in stress testing and will go for production.

            Already migrating from JBOSS 5 to Wildfly 8 took long time. We can't suddenly go for higher version after releasing stable 8.2.1-Final version.

             

            We found wildfly unable to handle concurrent http request even though undertow claim, it can handle 1 million connection.

            I think undertow developer can provide some hints to handle this.

             

            Hope you understand this situation and provide some more inputs to scaling wildfly server.

             

            Regards,

            Tarique

            1 of 1 people found this helpful
            • 3. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
              Tomaz Cerar Master

              Mohd tarique alam Khan wrote:


               

              We found wildfly unable to handle concurrent http request even though undertow claim, it can handle 1 million connection.

              I think undertow developer can provide some hints to handle this.

              There ware quite few performance related issues found in 8.x and fixed in later releases, that is where Jaikiran's suggestion to use later version comes from.

              Undertow can handle lots of concurrent connections/requests but it usually boils down to application that is running, so problem could not be related to web server at all.

               

              Did you do anything to identify hot spots with tools like jprofiler/yourkit to see where most cpu time is spent?

              Is it in web server related code or somewhere in your application?

               

              migrating from jboss 5 to 8 is a big change and there could be lots of other factors that could impact results you are seeing, web server is just one component in this whole stack.

              • 4. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                Mohd tarique alam Khan Newbie

                Resposnetime_communicationError.png

                See the new graph...it's too ugly to show the performance of Wildfly. Could you please provide the some workaround to handle this scenario?

                • 5. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                  Wolfgang Mayer Master

                  Did you make sure that the issue is not caused by memory leaks?

                  • 6. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                    Mohd tarique alam Khan Newbie

                    We don't have memory leaks and server is running without any JVM related exceptions.

                    • 7. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                      Wolfgang Mayer Master

                      Here I was rather refering to the Java memory (mainly the heap) and GC. 'Leak Suspects' do not neccessarily throw Java exceptions. I assume you did also upgrade the Java version (a good overview to the memory health may provide the Eclipse Memory Analyzer or even JConsole).

                      Could you also provide more details which tool you are using,  which figure you are exactly referring and how this figure changed compared to JBOSS 5.1. What elese did change in your environment?

                      Is this a special application where the issue appears? When speaking about HTTP I assume you are referring to a Web application (HTML page). Did you also try to localise the root cause via your browser (eg firebug)?

                      • 8. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                        Mohd tarique alam Khan Newbie

                        Thanks Wolfgang,

                         

                        Yes we are using JDK8_66 version. Also we have analyzed the heap dump using Eclipse memory analyzer but not found any GC leak suspects(We will go more deeply after 24 hour run) but we short run we have not found any leak suspects.

                        In earlier JBOSS 5.1, We were using JDK7. Now in new environment, I have provided all the information on my first post.

                        We are not generating HTML page. Our application is return XML response.

                        We are using JMeter through which we send http get or Post request to get XML response. Please see below log where you can see server stop responding and apache http client (version 4.3.6) sends connection time out after 70 sec(we have defined in our configuration). Also you can free pool status of httpclient.

                         

                        2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 4

                        2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

                        2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 6

                        2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

                        2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 15

                        2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

                        2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 5

                        2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

                        2016-05-18 14:21:14,652 INFO  [default task-257][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /proxy/uusd - Time taken for Request, /proxy/uusd/getOnTopFavourites : 26ms.

                        2016-05-18 14:22:24,841 ERROR [default task-74][be.belgacom.tv.brq.helpers.BrqExceptionHandler] [BRQHbrqh181421991] 150001:Communication failed to the proxy.

                        2016-05-18 14:22:24,867 INFO  [default task-74][be.belgacom.tv.common.BtaErrorMessageResolver] [BRQHbrqh181421991] Old Error Code :150001 New Error Code: 150001 Message: 0xbe100000:System Error.

                        2016-05-18 14:22:24,937 INFO  [default task-74][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /broker/bta - Time taken for Request, /broker/bta/getOnTopFavourites : 70425ms.

                         

                         

                        Regards,

                        Tarique

                        • 9. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                          Mohd tarique alam Khan Newbie

                          Thanks Tomaz,

                          Please see below log where you can see server stop responding and apache http client (version 4.3.6) sends connection time out after 70 sec(we have defined in our configuration). Also you can free pool status of httpclient.

                           

                          2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 4

                          2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

                          2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 6

                          2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-48][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

                          2016-05-18 14:21:14,622 INFO [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Available idle persistent connections stats.getAvailable() ::: 15

                          2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of connection requests being blocked awaiting a free connection stats.getPending() ::: 0

                          2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Number of persistent connections tracked by the connection manager currently being used to execute requests stats.getLeased() ::: 5

                          2016-05-18 14:21:14,622 INFO  [taskExecutor-brqh-77][be.belgacom.tv.http.common.BtaPoolingHttpClientConnectionManager] [BRQHbrqh181421991] Maximum number of allowed persistent connections stats.getMax() ::: 800

                          2016-05-18 14:21:14,652 INFO  [default task-257][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /proxy/uusd - Time taken for Request, /proxy/uusd/getOnTopFavourites : 26ms.

                          2016-05-18 14:22:24,841 ERROR [default task-74][be.belgacom.tv.brq.helpers.BrqExceptionHandler] [BRQHbrqh181421991] 150001:Communication failed to the proxy.

                          2016-05-18 14:22:24,867 INFO  [default task-74][be.belgacom.tv.common.BtaErrorMessageResolver] [BRQHbrqh181421991] Old Error Code :150001 New Error Code: 150001 Message: 0xbe100000:System Error.

                          2016-05-18 14:22:24,937 INFO  [default task-74][be.belgacom.tv.common.interceptor.RequestTimeInterceptor] [BRQHbrqh181421991] /broker/bta - Time taken for Request, /broker/bta/getOnTopFavourites : 70425ms.

                           

                           

                          Regards,

                          Tarique

                          • 10. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                            Tomaz Cerar Master

                            Hey,

                             

                            all you have is your client benchmark results.

                             

                            you would really need to do profiling of the java code on the server.

                             

                            get yourself profiler like Java Profiler - .NET Profiler - The profilers for Java and .NET professionals

                            for which you can get trial license. and connect it to running WildFly JVM and do CPU profiling while you are running benchmarks.

                            this will help identify where in the server / application code there is a bottleneck / hotspot.

                             

                            --

                            tomaz

                            • 11. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                              Wolfgang Mayer Master

                              Are you using a belgacom proxy server. If yes, did you verify that this issue is not related to belgacom?

                              Maybe you could also try Wireshark?

                              • 12. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                                Mohd tarique alam Khan Newbie

                                Thanks Wolfgang,

                                 

                                We have offshore develepment center for belgacom. We are not using any proxy for our application.

                                Belgacom application contains lots of proxy applications which communicate each other in intranet.

                                We checked with network team, there is no network issue as well. Today we are going to do profiling for our java application apart from thread and heap dump.

                                • 13. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                                  Mohd tarique alam Khan Newbie

                                  Hello Tomaz,

                                   

                                  We have done our testing but the result are the same. After some time (around after 2-3 minutes) with http load using JMeter, Undertow start giving read time out and stream closed error.

                                  Read Time out Error

                                  java.net.SocketTimeoutException: Read timed out

                                    at java.net.SocketInputStream.$$YJP$$socketRead0(Native Method) ~[?:1.8.0_66]

                                    at java.net.SocketInputStream.socketRead0(SocketInputStream.java) ~[?:1.8.0_66]

                                    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_66]

                                    at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_66]

                                    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_66]

                                    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:153) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) ~[httpcore-4.3.3.jar!/:4.3.3]

                                    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) ~[httpclient-4.3.6.jar!/:4.3.6]

                                    at be.belgacom.tv.http.common.BtaPoolingHttpClientManagerFactory.execute(BtaPoolingHttpClientManagerFactory.java:168) ~[bta-common.jar!/:?]

                                    at be.belgacom.tv.circuitbreaker.wrappers.BtaCBPoolingHttpClientManagerFactory.execute(BtaCBPoolingHttpClientManagerFactory.java:151) [circuitbreaker.jar:?]

                                    at be.belgacom.tv.brq.services.common.BrqHttpService.forwardHttpGetRequest(BrqHttpService.java:354) [classes:?]

                                    at be.belgacom.tv.brq.services.common.BrqHttpService.processHttpRequest(BrqHttpService.java:547) [classes:?]

                                    at be.belgacom.tv.brq.services.impl.AuthenticationServiceBeanImpl.authenticate(AuthenticationServiceBeanImpl.java:68) [classes:?]

                                    at be.belgacom.tv.brq.interceptor.AuthenticationInterceptorImpl.preHandle(AuthenticationInterceptorImpl.java:120) [classes:?]

                                    at org.springframework.web.servlet.HandlerExecutionChain.applyPreHandle(HandlerExecutionChain.java:134) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

                                    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:954) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

                                    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

                                    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

                                    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

                                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]

                                    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843) [spring-webmvc-4.1.7.RELEASE.jar!/:4.1.7.RELEASE]

                                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar!/:1.0.0.Final]

                                    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at be.belgacom.tv.http.filters.BtaRequestCompressionFilter.doFilter(BtaRequestCompressionFilter.java:88) [http-compression.jar!/:?]

                                    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) [wildfly-undertow-8.2.1.Final.jar!/:8.2.1.Final]

                                    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) [wildfly-undertow-8.2.1.Final.jar!/:8.2.1.Final]

                                    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:248) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:77) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:167) [undertow-servlet-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:761) [undertow-core-1.1.8.Final.jar!/:1.1.8.Final]

                                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_66]

                                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_66]

                                    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66]

                                   

                                  Stream Closed Error

                                   

                                  2016-05-24 13:10:37,436 ERROR [default task-341][be.belgacom.tv.brq.helpers.BrqExceptionHandler] [null] java.io.IOException: UT010029: Stream is closed

                                  at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:136)

                                  at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)

                                  at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)

                                  at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)

                                  at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)

                                  at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)

                                  at java.io.Writer.write(Writer.java:157)

                                  at org.springframework.util.StreamUtils.copy(StreamUtils.java:105)

                                  at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:109)

                                  at org.springframework.http.converter.StringHttpMessageConverter.writeInternal(StringHttpMessageConverter.java:40)

                                  at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:208)

                                  at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:161)

                                  at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:101)

                                  at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:185)

                                  at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:71)

                                  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:126)

                                  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:776)

                                  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:705)

                                  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)

                                  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959)

                                  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893)

                                  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)

                                  at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)

                                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)

                                  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)

                                  at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

                                  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)

                                  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130)

                                  at be.belgacom.tv.http.filters.BtaRequestCompressionFilter.doFilter(BtaRequestCompressionFilter.java:88)

                                  at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)

                                  at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132)

                                  at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85)

                                  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

                                  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

                                  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

                                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

                                  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

                                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

                                  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

                                  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)

                                  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70)

                                  at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)

                                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

                                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

                                  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:261)

                                  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:248)

                                  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:77)

                                  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:167)

                                  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)

                                  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:761)

                                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                                  at java.lang.Thread.run(Thread.java:745)



                                  Also please find the CPU, Memory, Heap graph. We are not having any problem on that side.

                                  Memory Usage.jpgCPU usage.jpgPerformance Usage.jpgSummary report.jpg

                                  Similar kind of error was found in Webservice with wildfly 8.2.

                                  http://stackoverflow.com/questions/29303854/wildfly-8-2-undertow-read-time-out

                                   

                                  Could you please provide more hint to handle this in undertow?

                                  P.S. We have tried with read time out attribute as well but result is same i.e.

                                  <http-listener name="default" socket-binding="http" max-buffered-request-size="65536" tcp-keep-alive="true" read-timeout="60000"/>

                                   

                                  Regards,

                                  Tarique

                                  • 14. Re: Performance degrades in Wildfly 8.2.1 compare to JBOSS 5.1 under heavy http loads
                                    Mohd tarique alam Khan Newbie

                                    Hello Everyone,

                                     

                                    We are able to solve this problem and it's good to share information on the forum.

                                    1 . Our server was not tuned to cater the maximum number of processes. To overcome this problem, we add the following lines to the /etc/security/limits.conf file:

                                    jboss soft nproc 8000

                                    jboss hard nproc 12000

                                    Remember we already did configuration to tune maximum file descriptors using below line in same /etc/security/limits.conf file

                                    jboss soft nofile 9999

                                    jboss hard nofile 9999

                                     

                                    Now after this configuration our server is able to create more threads

                                    <worker name="default" io-threads="20" task-max-threads="1024"/>

                                     

                                    2. Timeouts + OS time|close_waits on unused connections

                                    now the the very next "problem" we noticed under heavy load, was the high number of CLOSE|TIME_WAITS on OS level. Tuning http-listeners was solve this problem.

                                    <http-listener name="default" socket-binding="http" record-request-start-time="true" no-request-timeout="2500 request-parse-timeout="25000" tcp-keep-alive="true" read-timeout="30000"/>

                                    We able to solve this problem after <subsystem xmlns="urn:jboss:domain:undertow:1.2" statistics-enabled="true">

                                     

                                    3. Our component are also used for messaging so need to address messaging section as well.

                                    To overcome this problem, we added more element in  <subsystem xmlns="urn:jboss:domain:messaging:2.0"> (BOLD are new)

                                                   <persistence-enabled>true</persistence-enabled>

                                                    <thread-pool-max-size>10</thread-pool-max-size>

                                                    <security-enabled>false</security-enabled>

                                                   <statistics-enabled>true</statistics-enabled>

                                                    <journal-file-size>1024000</journal-file-size>

                                                    <journal-directory relative-to="jboss.server.config.dir" path="messagingjournal"></journal-directory>

                                                    <large-messages-directory relative-to="jboss.server.config.dir" path="messaginglargemessages"></large-messages-directory>

                                                    <bindings-directory relative-to="jboss.server.config.dir" path="messagingbindings"></bindings-directory>

                                                    <paging-directory relative-to="jboss.server.config.dir" path="paging"></paging-directory>

                                                    <journal-max-io>500</journal-max-io>

                                                    <journal-min-files>5</journal-min-files>

                                                    <journal-type>ASYNCIO</journal-type>

                                     

                                    Now our server is able to handle heavy load and able to achieve more TPS and thread than JBOSS5.1.

                                    Thanks you guys for your support and time.

                                     

                                    Regards,

                                    Tarique