Help: java.lang.RuntimeException: Operation failed with status WAITING

[s.horn]

Hello Community,

i have an urgend issue with JBoss 7.2.0.

I am running JBoss on server and about 1000 Clients connecting to the ejbs.

I am using the jbossEJBClient with PropertiesBasedEJBClientConfiguration:

Properties prop = new Properties();
prop.put("endpoint.name", "client-endpoint");
prop.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
prop.put("remote.connections", "default");
prop.put("remote.connection.default.port", "192.168.22.102");
prop.put("remote.connection.default.host", "4447");
prop.put("remote.connection.default.username", "*****");
prop.put("remote.connection.default.password", "*****");
prop.put("remote.connection.default.connect.timeout", "10000");
prop.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
prop.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
prop.put("remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
prop.put("remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL", "60000");

EJBClientConfiguration clientconfiguration = new PropertiesBasedEJBClientConfiguration(prop);

ContextSelector<EJBClientContext> ejbContextSelector = new ConfigBasedEJBClientContextSelector(clientconfiguration);
EJBClientContext.setSelector(ejbContextSelector);

EJBLocator<SessionRemote> locator = new StatelessEJBLocator<SessionRemote>(SessionRemote.class, "app, "module", "Session", "");
sessionRemote = EJBClient.createProxy(locator);

// login and work

...

// logout and disconnect use the following:

EJBClientContext.getCurrent().close();

This works like a charm half a day and suddenly nobody is able to connect to Jboss:

Everywhere the following Exception occurs:

WARN: Could not register a EJB receiver for connection to 192.168.22.102:4447
java.lang.RuntimeException: Operation failed with status WAITING
    at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93)
    at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75)
    at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
    at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:130)
    at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:100)
    at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:68)
    at de.procar.ecaros2.client.ClientSideServerConnection.getEcarosSessionRemoteInterface(ClientSideServerConnection.java:134)
    at de.procar.ecaros2.client.ServerConnection.connect(ServerConnection.java:496)
    at de.procar.ecaros2.client.basic.LoginPanel$1.doOnConstruct(LoginPanel.java:415)
    at de.procar.ecaros2.client.EcarosUtilities$2.construct(EcarosUtilities.java:484)
    at de.procar.swing.SwingWorker$2.run(SwingWorker.java:121)
    at java.lang.Thread.run(Thread.java:744)

Our users are very upset, cause the system crashes at least once a day.

I have no idea what could be the problem.

Please give me any hints.

Thanks

Sascha

Edit:

I forgot to mention that the server never stops running.

I am able to connect with the Cli.

Looking up the ejbs from serverside, for example from an Servlet of an other deployment in same Jboss is still working.

Only the remotelookups dont.

Next edit:

Found the following exception in server.log.

This occurs nearly at the moment when the problems start.

Is it possible that this is the root of the issue:

12:36:43,910 ERROR [org.jboss.as.ejb3] (Remoting "Server" task-2) JBAS014140: Schließe Channel Channel ID 2d1670cc (inbound) of Remoting connection 59bed0b1 to null aufgrund eines Fehlers: java.io.IOException: JBAS014560: Konnte Nachrichten-Outputstream für

        at org.jboss.as.ejb3.remote.EJBRemoteConnectorService.sendVersionMessage(EJBRemoteConnectorService.java:171)

        at org.jboss.as.ejb3.remote.EJBRemoteConnectorService.access$000(EJBRemoteConnectorService.java:73)

        at org.jboss.as.ejb3.remote.EJBRemoteConnectorService$ChannelOpenListener.channelOpened(EJBRemoteConnectorService.java:211)

        at org.jboss.remoting3.spi.SpiUtils$ServiceOpenTask.run(SpiUtils.java:126) [jboss-remoting-3.2.16.GA-redhat-1.jar:3.2.16.GA-redhat-1]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

Caused by: org.jboss.remoting3.NotOpenException: Writes closed

        at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108) [jboss-remoting-3.2.16.GA-redhat-1.jar:3.2.16.GA-redhat-1]

        at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297) [jboss-remoting-3.2.16.GA-redhat-1.jar:3.2.16.GA-redhat-1]

        at org.jboss.as.ejb3.remote.protocol.versionone.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:68)

        at org.jboss.as.ejb3.remote.EJBRemoteConnectorService.sendVersionMessage(EJBRemoteConnectorService.java:169)

        ... 6 more

[wdfink]

What you mean by login/logout.

Also I don't understand why you close the EJBClientContext, this should not be done except you register a new selector.

[s.horn]

Sorry,

login/logout...

Our users have to log in seperately over our application.

All i want to say with this line is, that they are able to do their normal work within our application, until jboss do not accept any further remote communication.

We create the clientcontext on every login to our application, therefor we must close it when the user logout.

Otherwise the used connections will not be closed and grow until too much files are open an server side...

Our users have the manner to just logout and do not close the application complete....

[wdfink]

If you keep the client open you should set the context once without closing it and only do lookups.

Other approach is to use the scoped context (see this doc Scoped EJB client contexts) which is complete under application control. In that case you create the InitialContext with properties, use it and close it.

I suppose this will work better with your approach.

[s.horn]

Hello Wolf-Dieter,

thank you for your reply.

We tried using Scoped EJB Client Context.

Our Problem is that we use Beans which return other Bean Instances.

When we try to invoke these Beaninstances there is not the correct ejb Context connected and we receive no EJB Receiver available...

I posted this issue within an other Discussion.

Last night we changed our lookup mechanism. We now use jboss-ejb-client.properties with JNDI Lookup with following properties.

remote.connections=default
remote.connection.default.host=192.168.98.111
remote.connection.default.port=4447
remote.connection.default.username=******
remote.connection.default.password=*******
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS=JBOSS-LOCAL-USER
remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=60000
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false

java.naming.factory.url.pkgs=org.jboss.ejb.client.naming

But again JBoss stoped acception any new remote connections after about 6 Hours running like a charm.

I found many errors like this in server.log

14:16:03,444 ERROR [org.jboss.remoting.remote.connection] (Remoting "procar-app2" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

They occure all the day.

May this failed connections be the cause of the JBoss crash?

Is it possiple to monitor the remoting-connector by cli?

Perhaps there are too many open remoting worker tasks?

Tonight we update JBoss from EAP 6.1 to EAP 6.2 in hope that the problem is already solved an we just did not found the jira issue...

[wdfink]

Could you explain what you mean by

Our Problem is that we use Beans which return other Bean Instances.

As I understand you use a standalone client.

[s.horn]

I tried to explain the problem here:

java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName, moduleName, distinctName] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@785c

We use a standalone client, but we use an single EJB as entry point to the server.

This ejb looks up to other ejb and returns them back to the client.

The client invokes methods of the returned ejb Instances.

Using scoped context do not work with this structure, because the ejbcontext is not passed throug the entrypoint bean to the returned instances of the other beans.

But as i said, this is not the current problem, because if we use jboss-ejb-client.properties configuration this is working.

Our problem is a jboss, which is suddenly not accepting new remote connections.

Local lookups , for example lookup done by an servlet deployed on the same jboss still working...

We currently try to open a thread in redhat support center, but it is hard to reach anyone of the redhat sales team...

Our customers are more than upset, because the system crashes once a day and we have no idea what is the root of the issue...

[wdfink]

If you are using EAP you should open a case for that.

I think an example to reproduce will be helpful, atm I did not understand which EJB you are returning (whether it is another remote) and what exactly goes wrong.

If you return a ejb (proxy) this will be different to former releases, there is less intelligence in it and the EJBClientContext will do the connection work.

[s.horn]

Hello Wolf-Dieter,

the EJB through EJB thing is not urgent, because we found an solution.

But this thread is based on an other remoting problem.

We figured out some more information about our problem:

We have at least on e ejb method which produces on invoke the following Exception on the client side:

java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:ecaros2-server-application, moduleName:ecaros2-server, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@29ab27

    at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:735)

    at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)

    at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183)

    at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:208)

    at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:256)

    at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:265)

    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198)

    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)

    at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)

    at com.sun.proxy.$Proxy31.executeQuery(Unknown Source)

    at de.procar.ecaros2.client.cache.QueryModuleCache.executeQuery(QueryModuleCache.java:401)

    at de.procar.ecaros2.client.basic.QueryResultTableModel.<init>(QueryResultTableModel.java:306)

    at de.procar.ecaros2.client.basic.ResultComponent.setQuery(ResultComponent.java:819)

    at de.procar.ecaros2.client.basic.ResultPanel.quickSearch(ResultPanel.java:335)

    at de.procar.ecaros2.client.EcarosUtilities$3.doOnConstruct(EcarosUtilities.java:969)

    at de.procar.ecaros2.client.EcarosUtilities$2.construct(EcarosUtilities.java:484)

    at de.procar.swing.SwingWorker$2.run(SwingWorker.java:121)

    at java.lang.Thread.run(Unknown Source)

this exception occurs only on windows clients and not at every invocation.

Linux clients are not affected.

On server side no exceptions are thrown.

The result is, that the client is not usable anymore, which is not the main problem.

But on server side every failed invocation causes an Remoting task WAITING (on object monitor).

This blocks the Remoting Thread until the JBoss is restarted.

And the JBoss could only be stopped by killing the process:

Here the thread dump of on of the blocked threads:

"Remoting "future2012" task-3" prio=10 tid=0x00007f7398073800 nid=0xeca in Object.wait() [0x00007f73f3dfb000]

   java.lang.Thread.State: WAITING (on object monitor)

    at java.lang.Object.wait(Native Method)

    - waiting on <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

    at java.lang.Object.wait(Object.java:503)

    at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:106)

    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:126)

    at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:114)

    at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:143)

    - locked <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

    at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:161)

    - locked <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

    at org.jboss.remoting3.remote.OutboundMessage.close(OutboundMessage.java:244)

    - locked <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

    at org.jboss.as.ejb3.remote.protocol.versionone.ChannelAssociation.releaseChannelMessageOutputStream(ChannelAssociation.java:85)

    at org.jboss.as.ejb3.remote.protocol.AbstractMessageHandler.writeException(AbstractMessageHandler.java:118)

    at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:166)

    at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:160)

    at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:435)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

    at java.lang.Thread.run(Thread.java:744)

   Locked ownable synchronizers:

    - <0x00000000e6512010> (a java.util.concurrent.ThreadPoolExecutor$Worker)

We are running out of remoting task threads successivly.

I think there is a bug within jboss remoting3.

I keep on investigation and try to create an example.

This may take a few days.

[s.horn]

Now i am able to reproduce the issue.

attached you can find the sourcecode for an example bean and client.

The issue occures in combination JBoss 7.3.0 (Linux) and the ejb Client on an windows machine (Both running with JDK 1.7.0_51)

When running the client after a view successful invocations the following exception occurs

java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:remotingtest, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@19d136a
        at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:735)
        at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)
        at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183)
        at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:208)
        at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:256)
        at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:265)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198)
        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)
        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)
        at com.sun.proxy.$Proxy0.sendMyContainer(Unknown Source)
        at de.procar.remotetest.RemotingTestClient.<init>(RemotingTestClient.java:48)
        at de.procar.remotetest.RemotingTestClient.main(RemotingTestClient.java:20)

The clients invokes the method with an increasing Object in object count (See source code).

Suddently this kills the ejb receiver without any further exception than the no ejb receiver available exception.

When the client is executed several times, Jboss stops cancelling the connections.

The number of Remoting Task Threads in State Waiting (on object monitor) increases until all tasks are blocked and Jboss is not responding on remoting requests anymore.

Can anyone confirm that this is a Bug?

That would be very helpful.

A JBoss which is dying once a week because it is running out of remoting tasks is unusable.

Thanks in advance.

[skozhenkov]

Hi,

I am wondering if you've been able to find a solution to that, I have similar issue here.

Thank you.

[pmm]

You you using remote EJB3 with SSL? If so check for a deadlock

Deadlock in XNIO in EAP 6.1

[s.horn]

Hello,

@Sergey Kozenkov: We have not found a solution, yet.

Currently we are switching to EAP 6.3 to see if the changes in marshalling and remoting will solve the problem.

Meanwhile we extended the remoting threads to 16 and restart the server every time when at least 3 threads are blocked. (Sometime 2 or 3 times the week )

@ Philippe Marshall: We do not use EJB3 with SSL.

If the threads are blocked we analyse the threaddump and it never detects deadlocks. The issue must be somewhere else...