Remote JNDI connection not working after some time

[olli24]

Hi there,

we are using JBoss AS 7.1.1.Final "Brontes" with JDK 1.7.0_7 and Java clients with remote JNDI.

All needed remote JNDI settings are defined via jboss-ejb-client property file.

On Client side we use: JRE 1.6.0_33 32-Bit under Windows OS

Normally the clients can connect to JBoss7 without any problems.

But after some time the clients cannot connect. In the client console “Could not register a EJB receiver for connection” is logged.

2013-09-30 11:33:00.576 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.logging     - Logging Provider: org.jboss.logging.Log4jLoggerProvider

2013-09-30 11:33:00.596 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client     - JBoss EJB Client version 1.0.5.Final

2013-09-30 11:33:00.616 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - endpoint.create.options. has the following options {}

2013-09-30 11:33:00.616 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - remote.connectionprovider.create.options. has the following options {org.xnio.Options.SSL_ENABLED=>false}

2013-09-30 11:33:00.616 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - remote.connection.default.connect.options. has the following options {org.xnio.Options.SSL_STARTTLS=>false,org.xnio.Options.SASL_POLICY_NOANONYMOUS=>false}

2013-09-30 11:33:00.616 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - remote.connection.default.channel.options. has the following options {}

2013-09-30 11:33:00.616 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - Connection org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration$RemotingConnectionConfigurationImpl@134b07e successfully created for connection named default

2013-09-30 11:33:00.616 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - No clusters configured in properties

2013-09-30 11:33:00.646 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.xnio     - XNIO Version 3.0.3.GA

2013-09-30 11:33:00.656 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.xnio.nio - XNIO NIO Implementation Version 3.0.3.GA

2013-09-30 11:33:00.676 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.remoting     - JBoss Remoting version 3.2.3.GA

2013-09-30 11:33:00.706 DEBUG [Remoting "config-based-ejb-client-endpoint" read-1] org.xnio.nio     - Started channel thread 'Remoting "config-based-ejb-client-endpoint" read-1', selector sun.nio.ch.WindowsSelectorImpl@15fc40c

2013-09-30 11:33:00.706 DEBUG [Remoting "config-based-ejb-client-endpoint" write-1] org.xnio.nio - Started channel thread 'Remoting "config-based-ejb-client-endpoint" write-1', selector sun.nio.ch.WindowsSelectorImpl@e49dcd

2013-09-30 11:33:00.756 ERROR [Remoting "config-based-ejb-client-endpoint" read-1] org.jboss.remoting.remote.connection - JBREM000200: Remote connection failed: java.io.IOException: An existing connection was forcibly closed by the remote host

2013-09-30 11:33:00.766 WARN  [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector - Could not register a EJB receiver for connection to remote://xxxx01:9000

java.lang.RuntimeException: java.io.IOException: An existing connection was forcibly closed by the remote host

at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:91)

at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:121)

at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:78)

at org.jboss.ejb.client.EJBClientContext.<clinit>(EJBClientContext.java:77)

at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:120)

at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)

at $Proxy0.create(Unknown Source)

In the JBoss server.log is no ERROR at this time. This is the corresponding server.log:

2013-09-30 11:32:06,249 DEBUG [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] Recover DelegatingSession [session=ClientSessionImpl [name=766ee36c-1ab8-11e3-b3be-005056a70052, username=null, closed=false, factory = ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0, backupConfig=null], metaData=()]@7f1dba]

2013-09-30 11:32:06,264 DEBUG [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] Recover DelegatingSession [session=ClientSessionImpl [name=766ee36c-1ab8-11e3-b3be-005056a70052, username=null, closed=false, factory = ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0], discoveryGroupConfiguration=null], connectorConfig=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory?server-id=0, backupConfig=null], metaData=()]@7f1dba]

2013-09-30 11:33:02,658 DEBUG [org.apache.catalina.session.ManagerBase] Start expire sessions StandardManager at 1380533582658 sessioncount 0

2013-09-30 11:33:02,658 DEBUG [org.apache.catalina.session.ManagerBase] End expire sessions StandardManager processingTime 0 expired sessions: 0

2013-09-30 11:33:02,658 DEBUG [org.apache.catalina.session.ManagerBase] Start expire sessions StandardManager at 1380533582658 sessioncount 0

2013-09-30 11:33:02,658 DEBUG [org.apache.catalina.session.ManagerBase] End expire sessions StandardManager processingTime 0 expired sessions: 0

2013-09-30 11:33:02,673 DEBUG [org.apache.catalina.session.ManagerBase] Start expire sessions StandardManager at 1380533582673 sessioncount 0

2013-09-30 11:33:02,673 DEBUG [org.apache.catalina.session.ManagerBase] End expire sessions StandardManager processingTime 0 expired sessions: 0

2013-09-30 11:33:07,314 DEBUG [org.jboss.jca.core.connectionmanager.pool.idle.IdleRemover] Notifying pools, interval: 450000

For comparison this is the client console output if the connection to JBoss was successful:

2013-09-30 12:30:12.278 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.logging     - Logging Provider: org.jboss.logging.Log4jLoggerProvider

2013-09-30 12:30:12.309 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client     - JBoss EJB Client version 1.0.5.Final

2013-09-30 12:30:12.324 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - endpoint.create.options. has the following options {}

2013-09-30 12:30:12.340 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - remote.connectionprovider.create.options. has the following options {org.xnio.Options.SSL_ENABLED=>true}

2013-09-30 12:30:12.340 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - remote.connection.default.connect.options. has the following options {org.xnio.Options.SASL_POLICY_NOANONYMOUS=>false,org.xnio.Options.SSL_STARTTLS=>true}

2013-09-30 12:30:12.340 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - remote.connection.default.channel.options. has the following options {}

2013-09-30 12:30:12.340 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - Connection org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration$RemotingConnectionConfigurationImpl@f87e2e successfully created for connection named default

2013-09-30 12:30:12.340 DEBUG [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.ejb.client.PropertiesBasedEJBClientConfiguration - No clusters configured in properties

2013-09-30 12:30:12.340 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.xnio     - XNIO Version 3.0.3.GA

2013-09-30 12:30:12.356 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.xnio.nio     - XNIO NIO Implementation Version 3.0.3.GA

2013-09-30 12:30:12.371 INFO  [thread applet-WebTerminal.mtTerminalApplet-1] org.jboss.remoting     - JBoss Remoting version 3.2.3.GA

2013-09-30 12:30:12.387 DEBUG [Remoting "config-based-ejb-client-endpoint" read-1] org.xnio.nio - Started channel thread 'Remoting "config-based-ejb-client-endpoint" read-1', selector sun.nio.ch.WindowsSelectorImpl@195aad9

2013-09-30 12:30:12.387 DEBUG [Remoting "config-based-ejb-client-endpoint" write-1] org.xnio.nio     - Started channel thread 'Remoting "config-based-ejb-client-endpoint" write-1', selector sun.nio.ch.WindowsSelectorImpl@144e463

2013-09-30 12:30:12.543 DEBUG [Remoting "config-based-ejb-client-endpoint" task-4] org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver - Channel Channel ID e7154574 (outbound) of Remoting connection 003035a1 to xxxx01/10.1.2.81:9000 opened for context EJBReceiverContext{clientContext=org.jboss.ejb.client.EJBClientContext@e97bef, receiver=Remoting connection EJB receiver [connection=Remoting connection <5c427e>,channel=jboss.ejb,nodename=oliver2]} Waiting for version handshake message from server

The difference between working and not working is the ERROR "JBREM000200: Remote connection failed: java.io.IOException: An existing connection was forcibly closed by the remote host" after the [write-1] output.

Who is closing the connection, JBoss or the client, and why?

Has JBoss a limitation for max socket connections?

What could be the reason for this problem, a server side JDK or client side JRE problem?

Thanks in advance for any help.

Oliver

[will.tatam]

Did all client then simultaneously fail to connect from there on until the server was restarted or did it only effect some users ?

Using an unpatched version of 7.1.3 we found that if a hibernate exception was passed back the the client and the client did not have the required exception class, then the client would moan about NoClassDef, but the server side component of the EJB client would be "killed" and no further invocations would be possible.

I would strongly advise against attempting to do any remote ejb stuff with anything less than AS 7.2

[olli24]

If the problem occurs, no client can connect until JBoss is restarted.

Then it works again for a day.

[jaikiran]

It's not JBoss or the client which is closely the connection. It's the "remote host" (the server's underlying OS). Search the net for "An existing connection was forcibly closed by the remote host" and you'll find some useful discussions.

[olli24]

Ok, you mean it’s related somehow to the OS. But what about following bug report?

https://issues.jboss.org/browse/EJBCLIENT-89?jql=text%20~%20%22ejb%20remote%20closed%22

This bug sounds like my problem.

I cannot understand why after JBoss restart the clients can connect again.

Also strange is, that we faced this problem only with client java 1.6 and not with client java 1.7. Of course this could be by fluke.

[olli24]

I have searched the net, but I didn’t find any really helpful information.

But we faced another may be useful information.

If the EJB Clients cannot connect to JBoss anymore  “Could not register a EJB receiver for connection to remote”, even a telnet on the JBoss remote port is not possible.

After JBoss restart everything is fine again for some time. Because this it seems, something JBoss/JDK internally is not working anymore.

Should the JBoss underlying JDK updated (currently we use JDK 1.7.0_7 under Windows OS)?

Is the client side JRE 1.6 causing this problem?

Thanks for any hints.

[jaikiran]

Oliver Katzer wrote:

even a telnet on the JBoss remote port is not possible.

If even telnet isn't working then it's a sign that something wrong on the server side. There can be various factors which might affect this, ranging from port not being open to firewall or other such networking details. You'll have to check with your network admin to see if they have an idea of what's wrong.

[olli24]

Well, firewall and network couldn’t actually the problem, because the connection to JBoss (EJB Client and telnet) is immediately working, after JBoss restart.

So, the networking configuration has not been changed.

Furthermore, on the same server we used before Sybase EAServer and never had this problem.

This connection problem occurs for the first time, after changing to JBoss.

This connection problem occurs on a few different production systems and not only on one.

[olli24]

The problem is independent from the clients JRE. It occurs also with JRE 1.7 on client side.

On one customer system it happens once a month, that JBoss closes all existing remote EJB connections and no new remote EJB connections are possible. In this situation it’s needed to restart the JBoss.

On another system, it happens more often.

JBoss is running on Windows OS as a service via jbosssvc.exe.

For me it looks like a bug in JBoss AS 7.1.1 final.

Is it known, that upgrading to eap-6.1.0.Alpha would help?

Thanks.

[olli24]

Our system is now working with EAP 6.1.0.Alpha1 (AS 7.2.0).

The problem does not occur so far, but we are afraid that the problem can happen again.

Because this I have following question:

Can someone of the JBoss developers confirm that something was fixed from AS 7.1 to AS 7.2 in the remote JNDI communication part that belongs to this problem?

We use JBoss under Windows as service via "jboss-native-2.0.10-windows".

Is it known if this service wrapper could cause problems?

Thanks,

Oliver

[lluporini]

Hi Oliver,

We're experiencing similar problems with AS 7.1.1

How did it go when you upgraded to AS 7.2?

Thanks,

Luis