7 Replies Latest reply on Jun 4, 2014 7:39 AM by xavier.baude

    WildFly 8.0.0.Final still leaking sockets with HTTPS?

    shuminli

      The issue I am about to report is very similar to Is WildFly leaking sockets when behind an HTTP proxy like nginx or lighttpd?. The difference is that was for HTTP, instead of HTTPS.  Bernd Nigmann reported HTTPS was fixed, but HTTPS socket leaking is still happening.

       

      The OS is "Red Hat Enterprise Linux Server release 6.5 (Santiago)".  JDK is "Java(TM) SE Runtime Environment (build 1.7.0_45-b18)".  WildFly is "wildfly-8.0.0.Final".

       

      Here are the steps to reproduce the issue.

       

      1. Install wildfly-8.0.0.Final.  Everything in the subsequent steps assume the path relative to the installation.

      2. Generate a self-signed certificate using Java's keytool following "Pure Java SSL-Setup using keytool" at "SSL setup guide - WildFly 8 - Project Documentation Editor".  You have to use a real host name etc.  For example, "foo.acme.com" is used as the name; "foo" as alias, "secret" as the password, "foo.keystore" as the keystore as in the document.  Save foo.keystore to standalone/configuration/

      3. Modify standalone/configuration/standalone.xml by adding following to the end of inside server/management/security-realms/

              <security-realm name="UndertowRealm">
                  <server-identities>
                      <ssl protocol="TLSv1">
                          <keystore path="foo.keystore" relative-to="jboss.server.config.dir" keystore-password="secret" alias="longus" key-password="secret"/>
                      </ssl>
                  </server-identities>
              </security-realm>

      4. Modify standalone/configuration/standalone.xml by adding following after server/profile/<subsystem xmlns="urn:jboss:domain:undertow:1.0">/server/http-listener

                          <https-listener name="https" socket-binding="https" security-realm="UndertowRealm"/>

      5. Start WildFly

                          ./bin/standalone.sh -b 0.0.0.0 -Djboss.bind.address.management=0.0.0.0

      Here is the output

       

      =========================================================================

       

        JBoss Bootstrap Environment

       

        JBOSS_HOME: /opt/local/wf

       

        JAVA: java

       

        JAVA_OPTS:  -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true

       

      =========================================================================

       

      21:24:36,897 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.0.Final

      21:24:37,380 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.0.Final

      21:24:37,525 INFO  [org.jboss.as] (MSC service thread 1-4) JBAS015899: WildFly 8.0.0.Final "WildFly" starting

      21:24:39,408 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

      21:24:39,435 INFO  [org.xnio] (MSC service thread 1-1) XNIO version 3.2.0.Final

      21:24:39,448 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.2.0.Final

      21:24:39,525 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 33) JBAS010280: Activating Infinispan subsystem.

      21:24:39,581 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 41) JBAS011800: Activating Naming Subsystem

      21:24:39,598 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 4.0.0.Final

      21:24:39,616 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 46) JBAS013171: Activating Security Subsystem

      21:24:39,617 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-3) JBAS010408: Starting JCA Subsystem (IronJacamar 1.1.3.Final)

      21:24:39,654 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 28) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)

      21:24:39,662 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 39) JBAS012615: Activated the following JSF Implementations: [main]

      21:24:39,683 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 50) JBAS015537: Activating WebServices Extension

      21:24:39,742 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013170: Current PicketBox version=4.0.20.Final

      21:24:39,752 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 49) JBAS017502: Undertow 1.0.0.Final starting

      21:24:39,760 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) JBAS010417: Started Driver service with driver-name = h2

      21:24:39,776 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) JBAS017502: Undertow 1.0.0.Final starting

      21:24:39,856 INFO  [org.jboss.as.naming] (MSC service thread 1-4) JBAS011802: Starting Naming Service

      21:24:39,858 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) JBAS015400: Bound mail session [java:jboss/mail/Default]

      21:24:40,487 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 49) JBAS017527: Creating file handler for path /opt/local/wf/welcome-content

      21:24:40,543 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017525: Started server default-server.

      21:24:40,570 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) JBAS017531: Host default-host starting

      21:24:40,845 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) JBAS017519: Undertow HTTP listener default listening on /0.0.0.0:8080

      21:24:41,366 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) JBAS015012: Started FileSystemDeploymentService for directory /opt/local/wf/standalone/deployments

      21:24:41,383 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]

      21:24:41,658 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) JBAS017519: Undertow HTTPS listener https listening on /0.0.0.0:8443

      21:24:41,934 INFO  [org.jboss.ws.common.management] (MSC service thread 1-2) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.2.3.Final

      21:24:42,015 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://0.0.0.0:9990/management

      21:24:42,015 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://0.0.0.0:9990

      21:24:42,016 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: WildFly 8.0.0.Final "WildFly" started in 5681ms - Started 188 of 237 services (82 services are lazy, passive or on-demand)

       

      6. Open another terminal and execute following curl to get the home page.

       

      curl -k https://foo.acme.com:8443

       

      or

       

      curl -k https://localhost:8443

       

      7. use lsof to see the leaking sockets.  Here is the result after 3 curl access to home page.  Each access generates one leaking socket.

       

      sudo lsof -u jboss|grep java |grep "identify protocol"

      java      23509 jboss  316u  sock                0,6      0t0 2610197115 can't identify protocol

      java      23509 jboss  317u  sock                0,6      0t0 2610197269 can't identify protocol

      java      23509 jboss  318u  sock                0,6      0t0 2610197336 can't identify protocol

       

      Please help to resolve the issue.  Thank you!

        • 1. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
          swd847

          I can't reproduce this. Do the sockets disappear after a little while? If so about how long?

          • 2. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
            swd847

            Also do you see any CPU activity after the curl is complete?

            • 3. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
              shuminli

              The the socket stays until WildFly is shutdown.  I have a while loop for curl since 10:25 AM EST.  The leaking socket count is over 4000 now.  The CPU does not show much of activities.  Thread count is going up steadily.

               

              What OS are you trying to reproduce?  I might be able to try it out at my end.

               

              Thanks.

              • 4. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
                shuminli

                After about one and half hours continuing curl, WildFly gave up with Memory error (different issue).  I guess WildFly was hit too hard and too quickly.  The main concern is the leaking sockets.  The number of leaking sockets stays at 9099 up to now.

                 

                Here is the error from client side:

                 

                curl: (28) Operation timed out after 300247 milliseconds with 0 out of 0 bytes received

                 

                Here is the error from server side:

                 

                12:03:00,091 ERROR [org.xnio.listener] (default I/O-2) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory

                        at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_45]

                        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_45]

                        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_45]

                        at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:57) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.BufferAllocator$2.allocate(BufferAllocator.java:55) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:149) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ssl.JsseSslConduitEngine.<init>(JsseSslConduitEngine.java:143) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ssl.JsseSslStreamConnection.<init>(JsseSslStreamConnection.java:71) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:45) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ssl.JsseAcceptingSslStreamConnection.accept(JsseAcceptingSslStreamConnection.java:37) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ssl.AbstractAcceptingSslChannel.accept(AbstractAcceptingSslChannel.java:187) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ChannelListeners$DelegatingChannelListener.handleEvent(ChannelListeners.java:1092) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) [xnio-nio-3.2.0.Final.jar:3.2.0.Final]

                        at org.xnio.nio.WorkerThread.run(WorkerThread.java:531) [xnio-nio-3.2.0.Final.jar:3.2.0.Final]

                • 5. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
                  xavier.baude

                  Hello Shumin,

                   

                  Have you find a solution ? I'm experiencing the same issue on a server.

                   

                  Wildfly 8.0.0 FInal

                   

                  Thanks

                  • 6. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
                    ctomc

                    Guys try with 8.1.0.Final there ware lots of improvements / bug fixes in this area.

                    • 7. Re: WildFly 8.0.0.Final still leaking sockets with HTTPS?
                      xavier.baude

                      Hi,

                       

                      Problem is solved with the release Wildfly 8.1.0 Final.

                       

                      This question is solve.

                       

                      Thanks.