2 Replies Latest reply on Oct 6, 2016 1:35 PM by spatwary04

    WildFly 9.0.1 EJB remoting causes socket leaks

    Arto Huusko Newbie

      In WildFly 9.0.1, incoming remote EJB invocations cause the server to leak sockets when the remoting connection is closed by the client.

       

      The sockets are visible in netstat output for a short while in TIME_WAIT status and then they go away, but the sockets can be seen in lsof output for the server process. This does not happen on every connection close, but it still happens much too often.

       

      The leak occurs when remoting is configured using <connector> and when configured using <http-connector>. In both configurations I had SSL enabled, I have not tested without SSL.

       

      More detailed description:

      • EJB client application is short lived process that does an EJB invocation and then exists.
      • Network connection over localhost
      • Before exit the connection is closed using EJBClientContext.getCurrent.close. Whether the close is done or not has no effect. (By the way, a separate issue: If the close is not done, the shutdown hook installed by the ejb client code sometimes dead locks and the process does not exit)
      • Result: netstat ALWAYS shows at least one end of the connection in TIME_WAIT status (it varies which end stays in TIME_WAIT), sometimes both ends.
      • Not all invocations, but many invocations produce this in lsof output of the server process:
      java14598 huuskoa  569u sock            0,8   0t0 2733585 protocol: TCP
      • Older linux kernel shows this as "can't identify protocol"
      • I also watched the traffic with tcpdump, and shutdown process varies almost on every connection
        • Sometimes both ends send FIN packet
        • Sometimes neither end sends FIN packet
        • Sometimes only end sends FIN
        • I tried to configure the client to use CLOSE_ABORT setting; as a result the client sometimes but very rarely sends RST

       

      Remoting is configured in wildfly like this:

      <subsystem xmlns="urn:jboss:domain:remoting:3.0">
          <endpoint worker="default"/>

          <http-connector name="http-remoting-connector" connector-ref="https1"

                          security-realm="RemotingRealm"/>

      </subsystem>

       

      EJB client context properties:

       

      callback.handler.class=xxxxxxx

       

      invocation.timeout=600000

       

      remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=true

       

      remote.connections=default

       

      remote.connection.default.host=localhost

      remote.connection.default.port=7443

      remote.connection.default.protocol=https-remoting

      remote.connection.default.connect.timeout=2000

      remote.connection.default.connect.options.org.xnio.Options.SSL_PROTOCOL=TLSv1.2

      remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false

      remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS=JBOSS-LOCAL-USER

       

      Java version 8u60.

       

      I tried to replace jboss-remoting in wildfly with JBoss Remoting 4.0.14.Final, but this did not help.

       

      Attached are two org.jboss.remoting TRACE logs: the first case did not leak a socket, the second case did leak.

        • 1. Re: WildFly 9.0.1 EJB remoting causes socket leaks
          Arto Huusko Newbie

          I spent a good while reading org.jboss.remoting and org.xnio traces, looking at the code and debugging it.

          Two changes to org.xnio code fixed the problem for me, but I am not sure if they are correct: [XNIO-254] SSL socket leak - JBoss Issue Tracker

           

          The way jboss remoting connection is closed seems to be very complicated, and there were many, many, many different log traces. This is most likely due to the non blocking and asynchronous nature of the connections.

           

          In most traces the socket is closed on the server because the server tries, as response to close request from client, to send a close request to the client, and the client has already exited, so write causes exception which closes the socket.

           

          That does not happen always. And when the timing on all the threads and the client process are just right, it seems that the server first closes the write end, and when the server closes the read end, then XNIO's JsseSslStreamSourceConduit actually does not do anything (terminateReads method that class never terminates the reads when tls=true).

          • 2. Re: WildFly 9.0.1 EJB remoting causes socket leaks
            spatwary04 Newbie

            Arto,

            we are using wildfly10 and facing similar issue. do you know what resolved your issue.

             

             

            shiva