Client Address In Use Issues In Client Terminal Server Environment
After upgrading from JBoss AS 4.2.2 to WildFly 9.0.2, some of our customers who use a client terminal server to host standalone client applications encounter exceptions on the client side when trying to connect to the server.
These exceptions indicate that an address in use and usually look something like this:
java.net.BindException: Address already in use: connect
There isn't much information beyond that and it doesn't tell what address is in use, though I am guessing it's referring to a local port on the client terminal server.
We also see "EJBCLIENT000025: No EJB receiver available..." but I suspect it's for the same reason.
Both seem to happen at random.
The server itself is working fine and can be connected to without these issues when not running the client applications on the terminal server.
Further details:
- The clients connect to a standalone server
- A majority of outbound client traffic to the server is remote EJB invocations, with a smaller amount of remote JMS and standard HTTP traffic.
- All EJB/JMS traffic goes to the server http port/http-remoting endpoint
- The Undertow http-listener has tcp-keep-alive on, though it has been observed with the setting on or off, but does NOT define read/write timeouts at this time. See this discussion for more about that Lots of Sockets Opened by Default I/O Threads?
- The EJB client API with scoped contexts is used on the client side. Most applications create a single context [a small number of additional contexts may be created for connecting to other servers] and use it for the lifetime of the application. A 60 second ...connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL is set on the client/context properties, but I would guess that it isn't especially useful when read/write timeouts are not configured on the server.
One difference I have noticed between JBoss AS 4 and WildFly 9 is that when the client applications are idle the socket resources for EJB client connections remain in the ESTABLISHED state, while on JBoss AS 4 they would end up in the CLOSE_WAIT state [there were more socket resources used by clients connecting to a JBoss AS 4 server]. I suppose WildFly EJB/JMS client traffic is now considered HTTP traffic, which may mean any configuration on client terminal server related to HTTP connections may need to be adjusted.
One question I do have is if the set of XNIO configuration options or other options that can be applied to EJB client connections extends beyond those described here Overview of Client properties - EJB Client Library (AS7+) - Project Documentation Editor
While I don't think idle EJB clients being in the established state is leading to port depletion on client terminal servers, I think it would be ideal if the socket resources on the client and server would be released when clients are idle anyway. Is there a way to configure the client or server so that the connections for idle clients are closed after a period of inactivity but the client context remains intact [i.e. close the network connections without closing the client context]? The closest thing I have been able to find is to define read/write timeouts on the http-listener, which did result in the behavior I was talking about, but as I mentioned earlier these aren't defined due to difficulties establishing an appropriate timeout because of varying application behaviors [some applications may execute long running EJB invocations where there is a period of read/write inactivity on the channel and this would result in the channel being closed before the server can send a response] .
I am thinking that this has to do more with the client terminal server component than the client application or the server, but I thought I would ask in case anyone has encountered this or has any ideas.
