7 Replies Latest reply on Apr 12, 2016 12:47 AM by jaikiran pai

    Too many intra-process localhost connections

    Luís M. Costa Newbie

      Hi!

       

      I'm observing many localhost connections. The number of connections keeps increasing leading wildfly to just log connection problems because Windows runs out of available ports:

       

      Log entry example

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-6) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

      2016-03-30 10:44:26,222 ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException

       

       

      Using netstat (and filtering by PID) I see a lot of these connections:

      TCP    127.0.0.1:61313        127.0.0.1:61314        ESTABLISHED     8592

        TCP    127.0.0.1:61314        127.0.0.1:61313        ESTABLISHED     8592

        TCP    127.0.0.1:61315        127.0.0.1:61316        ESTABLISHED     8592

        TCP    127.0.0.1:61316        127.0.0.1:61315        ESTABLISHED     8592

        TCP    127.0.0.1:61317        127.0.0.1:61318        ESTABLISHED     8592

        TCP    127.0.0.1:61318        127.0.0.1:61317        ESTABLISHED     8592

        TCP    127.0.0.1:61319        127.0.0.1:61320        ESTABLISHED     8592

        TCP    127.0.0.1:61320        127.0.0.1:61319        ESTABLISHED     8592

        TCP    127.0.0.1:61322        127.0.0.1:61323        ESTABLISHED     8592

        TCP    127.0.0.1:61323        127.0.0.1:61322        ESTABLISHED     8592

        TCP    127.0.0.1:61324        127.0.0.1:61325        ESTABLISHED     8592

        TCP    127.0.0.1:61325        127.0.0.1:61324        ESTABLISHED     8592

        TCP    127.0.0.1:61326        127.0.0.1:61327        ESTABLISHED     8592

        TCP    127.0.0.1:61327        127.0.0.1:61326        ESTABLISHED     8592

        TCP    127.0.0.1:61328        127.0.0.1:61329        ESTABLISHED     8592

        TCP    127.0.0.1:61329        127.0.0.1:61328        ESTABLISHED     8592

        TCP    127.0.0.1:61331        127.0.0.1:61332        ESTABLISHED     8592

        TCP    127.0.0.1:61332        127.0.0.1:61331        ESTABLISHED     8592

        TCP    127.0.0.1:61333        127.0.0.1:61334        ESTABLISHED     8592

        TCP    127.0.0.1:61334        127.0.0.1:61333        ESTABLISHED     8592

        TCP    127.0.0.1:61335        127.0.0.1:61336        ESTABLISHED     8592

        TCP    127.0.0.1:61336        127.0.0.1:61335        ESTABLISHED     8592

        TCP    127.0.0.1:61337        127.0.0.1:61338        ESTABLISHED     8592

        TCP    127.0.0.1:61338        127.0.0.1:61337        ESTABLISHED     8592

        TCP    127.0.0.1:61365        127.0.0.1:61366        ESTABLISHED     8592

        TCP    127.0.0.1:61366        127.0.0.1:61365        ESTABLISHED     8592

        TCP    127.0.0.1:61367        127.0.0.1:61368        ESTABLISHED     8592

        TCP    127.0.0.1:61368        127.0.0.1:61367        ESTABLISHED     8592

        TCP    127.0.0.1:61370        127.0.0.1:61371        ESTABLISHED     8592

        TCP    127.0.0.1:61371        127.0.0.1:61370        ESTABLISHED     8592

        TCP    127.0.0.1:61644        127.0.0.1:61645        ESTABLISHED     8592

        TCP    127.0.0.1:61645        127.0.0.1:61644        ESTABLISHED     8592

        TCP    127.0.0.1:61659        127.0.0.1:61660        ESTABLISHED     8592

        TCP    127.0.0.1:61660        127.0.0.1:61659        ESTABLISHED     8592

        TCP    127.0.0.1:61717        127.0.0.1:61718        ESTABLISHED     8592

        TCP    127.0.0.1:61718        127.0.0.1:61717        ESTABLISHED     8592

       

       

      As we can see these are local intra-process connections.

       

      Setup:

      I've tried/observed this in:

      Windows Server 2012 and Windows 2007 Professional

      JDK 8u20 and JDK 8u77

      Wilfly 8.2.1-final

       

       

      Some facts:

      1)

      Just by starting a fresh Wildfly (just downloaded), and without any application deployed I see some of the connections:

        TCP    127.0.0.1:54808        127.0.0.1:54812        ESTABLISHED     11928

        TCP    127.0.0.1:54809        127.0.0.1:54811        ESTABLISHED     11928

        TCP    127.0.0.1:54810        127.0.0.1:54821        ESTABLISHED     11928

        TCP    127.0.0.1:54811        127.0.0.1:54809        ESTABLISHED     11928

        TCP    127.0.0.1:54812        127.0.0.1:54808        ESTABLISHED     11928

        TCP    127.0.0.1:54813        127.0.0.1:54814        ESTABLISHED     11928

        TCP    127.0.0.1:54814        127.0.0.1:54813        ESTABLISHED     11928

        TCP    127.0.0.1:54815        127.0.0.1:54816        ESTABLISHED     11928

        TCP    127.0.0.1:54816        127.0.0.1:54815        ESTABLISHED     11928

        TCP    127.0.0.1:54817        127.0.0.1:54818        ESTABLISHED     11928

        TCP    127.0.0.1:54818        127.0.0.1:54817        ESTABLISHED     11928

        TCP    127.0.0.1:54819        127.0.0.1:54820        ESTABLISHED     11928

        TCP    127.0.0.1:54820        127.0.0.1:54819        ESTABLISHED     11928

        TCP    127.0.0.1:54821        127.0.0.1:54810        ESTABLISHED     11928

        TCP    127.0.0.1:54822        127.0.0.1:54824        ESTABLISHED     11928

        TCP    127.0.0.1:54823        127.0.0.1:54825        ESTABLISHED     11928

        TCP    127.0.0.1:54824        127.0.0.1:54822        ESTABLISHED     11928

        TCP    127.0.0.1:54825        127.0.0.1:54823        ESTABLISHED     11928

        TCP    127.0.0.1:54826        127.0.0.1:54827        ESTABLISHED     11928

        TCP    127.0.0.1:54827        127.0.0.1:54826        ESTABLISHED     11928

        TCP    127.0.0.1:54828        127.0.0.1:54829        ESTABLISHED     11928

        TCP    127.0.0.1:54829        127.0.0.1:54828        ESTABLISHED     11928

      This value does not seem to increase when there are no deployed applications.

       

      2)

      When I deploy an application that creates some HTTP connections and also WebSockets the number of connections starts to grow.

       

      3)

      Using the debugger I has not able to catch how are the connections (sockets - I assume that server socket is being binded to port 0) being created .

       

      4)

      Using WinCap to capture 127.0.0.1 traffic I've found the following pattern:

       

      Client Connects [SYN]

      Server Ack [SYN, ACK]

      Client Ack [ACK]

      Client Push [PSH] - It pushes 8 bytes (e.g., in hex, e6202552f2b95f6f)

      Optionally the server pushes some data periodically (always the same)

      Server Push [PSH] - It pushes 1 bytes (in hex, 1)

       

      Does someone has any idea on what the problem may be? Or give any additional troubleshooting tip?

       

      Thanks,

      Luís M. Costa