4 Replies Latest reply on Dec 15, 2017 12:59 PM by jfisherdev

    Client Address In Use Issues In Client Terminal Server Environment

    jfisherdev Novice

      After upgrading from JBoss AS 4.2.2 to WildFly 9.0.2, some of our customers who use a client terminal server to host standalone client applications encounter exceptions on the client side when trying to connect to the server.

       

      These exceptions indicate that an address in use and usually look something like this:

       

      java.net.BindException: Address already in use: connect

       

      There isn't much information beyond that and it doesn't tell what address is in use, though I am guessing it's referring to a local port on the client terminal server.

       

      We also see "EJBCLIENT000025: No EJB receiver available..." but I suspect it's for the same reason.

       

      Both seem to happen at random.

       

      The server itself is working fine and can be connected to without these issues when not running the client applications on the terminal server.

       

      Further details:

      - The clients connect to a standalone server

      - A majority of outbound client traffic to the server is remote EJB invocations, with a smaller amount of remote JMS and standard HTTP traffic.

      - All EJB/JMS traffic goes to the server http port/http-remoting endpoint

      - The Undertow http-listener has tcp-keep-alive on, though it has been observed with the setting on or off, but does NOT define read/write timeouts at this time. See this discussion for more about that Lots of Sockets Opened by Default I/O Threads?

      - The EJB client API with scoped contexts is used on the client side. Most applications create a single context [a small number of additional contexts may be created for connecting to other servers] and use it for the lifetime of the application. A 60 second ...connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL is set on the client/context properties, but I would guess that it isn't especially useful when read/write timeouts are not configured on the server.

       

      One difference I have noticed between JBoss AS 4 and WildFly 9 is that when the client applications are idle the socket resources for EJB client connections remain in the ESTABLISHED state, while on JBoss AS 4 they would end up in the CLOSE_WAIT state [there were more socket resources used by clients connecting to a JBoss AS 4 server]. I suppose WildFly EJB/JMS client traffic is now considered HTTP traffic, which may mean any configuration on client terminal server related to HTTP connections may need to be adjusted.

       

      One question I do have is if the set of XNIO configuration options or other options that can be applied to EJB client connections extends beyond those described here Overview of Client properties - EJB Client Library (AS7+) - Project Documentation Editor

      While I don't think idle EJB clients being in the established state is leading to port depletion on client terminal servers, I think it would be ideal if the socket resources on the client and server would be released when clients are idle anyway. Is there a way to configure the client or server so that the connections for idle clients are closed after a period of inactivity but the client context remains intact [i.e. close the network connections without closing the client context]? The closest thing I have been able to find is to define read/write timeouts on the http-listener, which did result in the behavior I was talking about, but as I mentioned earlier these aren't defined due to difficulties establishing an appropriate timeout because of varying application behaviors [some applications may execute long running EJB invocations where there is a period of read/write inactivity on the channel and this would result in the channel being closed before the server can send a response] .

       

      I am thinking that this has to do more with the client terminal server component than the client application or the server, but I thought I would ask in case anyone has encountered this or has any ideas.

        • 1. Re: Client Address In Use Issues In Client Terminal Server Environment
          Tomaz Cerar Master

          Is there any chance you can upgrade to WildFly 11 and its new ejb client?

          • 2. Re: Client Address In Use Issues In Client Terminal Server Environment
            jfisherdev Novice

            Unfortunately not at this time. Is there something in particular with the newer version that would help in this case?

             

            The main thing I am currently suspecting is socket address reuse. Would this be environment-specific or a client configuration setting? org.xnio.Options.REUSE_ADDRESSES appears to map to SocketOptions.SO_REUSEADDR, but I'm not sure if it applies to EJB client configuration. I'm thinking address reuse should be disabled in this case.

            • 3. Re: Client Address In Use Issues In Client Terminal Server Environment
              jaikiran pai Master

              jfisherdev  wrote:

               

               

               

              The main thing I am currently suspecting is socket address reuse.

              I know you did mention there's not much other than that BindException message, but just to be sure, there isn't an associated exception stracktrace either? Without that it will really hard to even guess what's triggering this.

               

              jfisherdev  wrote:

               

               

               

              org.xnio.Options.REUSE_ADDRESSES appears to map to SocketOptions.SO_REUSEADDR, but I'm not sure if it applies to EJB client configuration. I'm thinking address reuse should be disabled in this case.

              It does get used if it's set as a xnio option through the EJB client configuration properties.

              • 4. Re: Client Address In Use Issues In Client Terminal Server Environment
                jfisherdev Novice

                I have seen different errors depending on how the client was trying to communicate with the server, but I think they all point to the same issue.

                 

                Here's one from a JAX-WS client invocation [essentially the same thing with a simple HttpURLConnection]:

                 

                java.net.BindException: Address already in use: connect
                at java.net.DualStackPlainSocketImpl.connect0(Native Method)
                at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
                at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
                at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
                at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
                at java.net.PlainSocketImpl.connect(Unknown Source)
                at java.net.SocksSocketImpl.connect(Unknown Source)
                at java.net.Socket.connect(Unknown Source)
                at java.net.Socket.connect(Unknown Source)
                at sun.net.NetworkClient.doConnect(Unknown Source)
                at sun.net.www.http.HttpClient.openServer(Unknown Source)
                at sun.net.www.http.HttpClient.openServer(Unknown Source)
                at sun.net.www.http.HttpClient.<init>(Unknown Source)
                at sun.net.www.http.HttpClient.New(Unknown Source)
                at sun.net.www.http.HttpClient.New(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(Unknown Source)
                at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(Unknown Source)
                at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(Unknown Source)
                at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(Unknown Source)
                at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(Unknown Source)
                at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Unknown Source)
                at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Unknown Source)
                at com.sun.xml.internal.ws.client.Stub.process(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(Unknown Source)
                at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(Unknown Source)
                

                 

                 

                Here is one from a remote JMS client:

                 

                HQ214016: Failed to create netty connection
                java.net.SocketException: Address already in use: no further information: $HOST$/$IP$
                at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
                at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
                at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
                at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:281)
                at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
                at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
                at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
                at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
                at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
                at java.lang.Thread.run(Unknown Source)
                

                 

                Here is one from a remote EJB client:

                 

                java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:$app$, moduleName:$app-module$, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext
                at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:774)
                at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)
                at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
                at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
                at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
                at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
                at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
                

                 

                 

                I think what's happening is that there is a client application launch process that makes a lot of connections to the server HTTP port and then closes them. This leaves a lot of connections in the TIME_WAIT state. If address reuse is enabled, which is NOT a socket option being set to true by the WildFly EJB/Remoting/XNIO libraries by default [unless you tell it to] so I'm thinking it would be a behavior of the underlying OS, there is a good chance a TIME_WAIT connection to the server HTTP port would be reused, which would lead to an address in use exception. I think this always has been a potential issue in this type of environment. The reason it probably wasn't seen in the past was that most of the outbound client application traffic wasn't going to the server HTTP port, like it is now with WildFly, but the various ports that were used for different types of communication [EJB, JMS, RMI] on JBoss AS 4 so the chances of reusing a connection to the same endpoint were far less.

                 

                I'm thinking we will need to look at the launch process to see why a client is opening so many connections and then closing them after a short period of time as well as the client terminal server environment configuration.

                 

                I would still be interested to know more about EJB client configuration capabilities as I asked about.

                 

                Thank you.