12 Replies Latest reply on Aug 26, 2014 6:28 AM by s.horn

    Help: java.lang.RuntimeException: Operation failed with status WAITING

    s.horn Newbie

      Hello Community,

       

      i have an urgend issue with JBoss 7.2.0.

      I am running JBoss on server and about 1000 Clients connecting to the ejbs.

      I am using the jbossEJBClient with PropertiesBasedEJBClientConfiguration:

       

      Properties prop = new Properties();
      prop.put("endpoint.name", "client-endpoint");
      prop.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
      prop.put("remote.connections", "default");
      prop.put("remote.connection.default.port", "192.168.22.102");
      prop.put("remote.connection.default.host", "4447");
      prop.put("remote.connection.default.username", "*****");
      prop.put("remote.connection.default.password", "*****");
      prop.put("remote.connection.default.connect.timeout", "10000");
      prop.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
      prop.put("remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT", "false");
      prop.put("remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS", "JBOSS-LOCAL-USER");
      prop.put("remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL", "60000");

       

      EJBClientConfiguration clientconfiguration = new PropertiesBasedEJBClientConfiguration(prop);

      ContextSelector<EJBClientContext> ejbContextSelector = new ConfigBasedEJBClientContextSelector(clientconfiguration);
      EJBClientContext.setSelector(ejbContextSelector);

      EJBLocator<SessionRemote> locator = new StatelessEJBLocator<SessionRemote>(SessionRemote.class, "app, "module", "Session", "");
      sessionRemote = EJBClient.createProxy(locator);

       

      // login and work

      ...

      // logout and disconnect use the following:

      EJBClientContext.getCurrent().close();

       

      This works like a charm half a day and suddenly nobody is able to connect to Jboss:

      Everywhere the following Exception occurs:

       

      WARN: Could not register a EJB receiver for connection to 192.168.22.102:4447
      java.lang.RuntimeException: Operation failed with status WAITING
          at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:93)
          at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75)
          at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
          at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.setupEJBReceivers(ConfigBasedEJBClientContextSelector.java:130)
          at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:100)
          at org.jboss.ejb.client.remoting.ConfigBasedEJBClientContextSelector.<init>(ConfigBasedEJBClientContextSelector.java:68)
          at de.procar.ecaros2.client.ClientSideServerConnection.getEcarosSessionRemoteInterface(ClientSideServerConnection.java:134)
          at de.procar.ecaros2.client.ServerConnection.connect(ServerConnection.java:496)
          at de.procar.ecaros2.client.basic.LoginPanel$1.doOnConstruct(LoginPanel.java:415)
          at de.procar.ecaros2.client.EcarosUtilities$2.construct(EcarosUtilities.java:484)
          at de.procar.swing.SwingWorker$2.run(SwingWorker.java:121)
          at java.lang.Thread.run(Thread.java:744)

       

      Our users are very upset, cause the system crashes at least once a day.

      I have no idea what could be the problem.

       

      Please give me any hints.

       

      Thanks

       

      Sascha

       

      Edit:

      I forgot to mention that the server never stops running.

      I am able to connect with the Cli.

      Looking up the ejbs from serverside, for example from an Servlet of an other deployment in same Jboss is still working.

      Only the remotelookups dont.

       

      Next edit:

      Found the following exception in server.log.

      This occurs nearly at the moment when the problems start.

      Is it possible that this is the root of the issue:

       

      12:36:43,910 ERROR [org.jboss.as.ejb3] (Remoting "Server" task-2) JBAS014140: Schließe Channel Channel ID 2d1670cc (inbound) of Remoting connection 59bed0b1 to null aufgrund eines Fehlers: java.io.IOException: JBAS014560: Konnte Nachrichten-Outputstream für

       

       

              at org.jboss.as.ejb3.remote.EJBRemoteConnectorService.sendVersionMessage(EJBRemoteConnectorService.java:171)

              at org.jboss.as.ejb3.remote.EJBRemoteConnectorService.access$000(EJBRemoteConnectorService.java:73)

              at org.jboss.as.ejb3.remote.EJBRemoteConnectorService$ChannelOpenListener.channelOpened(EJBRemoteConnectorService.java:211)

              at org.jboss.remoting3.spi.SpiUtils$ServiceOpenTask.run(SpiUtils.java:126) [jboss-remoting-3.2.16.GA-redhat-1.jar:3.2.16.GA-redhat-1]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

              at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

      Caused by: org.jboss.remoting3.NotOpenException: Writes closed

              at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108) [jboss-remoting-3.2.16.GA-redhat-1.jar:3.2.16.GA-redhat-1]

              at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:297) [jboss-remoting-3.2.16.GA-redhat-1.jar:3.2.16.GA-redhat-1]

              at org.jboss.as.ejb3.remote.protocol.versionone.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:68)

              at org.jboss.as.ejb3.remote.EJBRemoteConnectorService.sendVersionMessage(EJBRemoteConnectorService.java:169)

              ... 6 more

        • 1. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
          Wolf-Dieter Fink Master

          What you mean by login/logout.

          Also I don't understand why you close the EJBClientContext, this should not be done except you register a new selector.

          • 2. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
            s.horn Newbie

            Sorry,

            login/logout...

            Our users have to log in seperately over our application.

            All i want to say with this line is, that they are able to do their normal work within our application, until jboss do not accept any further remote communication.

             

            We create the clientcontext on every login to our application, therefor we must close it when the user logout.

            Otherwise the used connections will not be closed and grow until too much files are open an server side...

             

            Our users have the manner to just logout and do not close the application complete....

            • 3. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
              Wolf-Dieter Fink Master

              If you keep the client open you should set the context once without closing it and only do lookups.

               

              Other approach is to use the scoped context (see this doc Scoped EJB client contexts) which is complete under application control. In that case you create the InitialContext with properties, use it and close it.

              I suppose this will work better with your approach.

              • 4. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                s.horn Newbie

                Hello Wolf-Dieter,

                 

                thank you for your reply.

                We tried using Scoped EJB Client Context.

                Our Problem is that we use Beans which return other Bean Instances.

                When we try to invoke these Beaninstances there is not the correct ejb Context connected and we receive no EJB Receiver available...

                I posted this issue within an other Discussion.

                 

                Last night we changed our lookup mechanism. We now use jboss-ejb-client.properties with JNDI Lookup with following properties.

                 

                remote.connections=default
                remote.connection.default.host=192.168.98.111
                remote.connection.default.port=4447
                remote.connection.default.username=******
                remote.connection.default.password=*******
                remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
                remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
                remote.connection.default.connect.options.org.xnio.Options.SASL_DISALLOWED_MECHANISMS=JBOSS-LOCAL-USER
                remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=60000
                remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false

                 

                 

                java.naming.factory.url.pkgs=org.jboss.ejb.client.naming

                 

                But again JBoss stoped acception any new remote connections after about 6 Hours running like a charm.

                 

                I found many errors like this in server.log

                14:16:03,444 ERROR [org.jboss.remoting.remote.connection] (Remoting "procar-app2" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

                They occure all the day.

                 

                May this failed connections be the cause of the JBoss crash?

                Is it possiple to monitor the remoting-connector by cli?

                Perhaps there are too many open remoting worker tasks?

                 

                Tonight we update JBoss from EAP 6.1 to EAP 6.2 in hope that the problem is already solved an we just did not found the jira issue...

                • 5. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                  Wolf-Dieter Fink Master

                  Could you explain what you mean by

                  Our Problem is that we use Beans which return other Bean Instances.

                  As I understand you use a standalone client.

                  • 6. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                    s.horn Newbie

                    I tried to explain the problem here:

                     

                    java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName, moduleName, distinctName] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@785c

                     

                    We use a standalone client, but we use an single EJB as entry point to the server.

                    This ejb looks up to other ejb and returns them back to the client.

                     

                    The client invokes methods of the returned ejb Instances.

                    Using scoped context do not work with this structure, because the ejbcontext is not passed throug the entrypoint bean to the returned instances of the other beans.

                     

                    But as i said, this is not the current problem, because if we use jboss-ejb-client.properties configuration this is working.

                     

                    Our problem is a jboss, which is suddenly not accepting new remote connections.

                    Local lookups , for example lookup done by an servlet deployed on the same jboss still working...

                     

                    We currently try to open a thread in redhat support center, but it is hard to reach anyone of the redhat sales team...

                     

                    Our customers are more than upset, because the system crashes once a day and we have no idea what is the root of the issue...

                    • 7. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                      Wolf-Dieter Fink Master

                      If you are using EAP you should open a case for that.

                      I think an example to reproduce will be helpful, atm I did not understand which EJB you are returning (whether it is another remote) and what exactly goes wrong.

                       

                      If you return a ejb (proxy) this will be different to former releases, there is less intelligence in it and the EJBClientContext will do the connection work.

                      • 8. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                        s.horn Newbie

                        Hello Wolf-Dieter,

                         

                        the EJB through EJB thing is not urgent, because we found an solution.

                         

                        But this thread is based on an other remoting problem.

                         

                        We figured out some more information about our problem:

                         

                        We have at least on e ejb method which produces on invoke the following Exception on the client side:

                        java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:ecaros2-server-application, moduleName:ecaros2-server, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@29ab27

                            at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:735)

                            at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)

                            at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183)

                            at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:208)

                            at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:256)

                            at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:265)

                            at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198)

                            at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)

                            at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)

                            at com.sun.proxy.$Proxy31.executeQuery(Unknown Source)

                            at de.procar.ecaros2.client.cache.QueryModuleCache.executeQuery(QueryModuleCache.java:401)

                            at de.procar.ecaros2.client.basic.QueryResultTableModel.<init>(QueryResultTableModel.java:306)

                            at de.procar.ecaros2.client.basic.ResultComponent.setQuery(ResultComponent.java:819)

                            at de.procar.ecaros2.client.basic.ResultPanel.quickSearch(ResultPanel.java:335)

                            at de.procar.ecaros2.client.EcarosUtilities$3.doOnConstruct(EcarosUtilities.java:969)

                            at de.procar.ecaros2.client.EcarosUtilities$2.construct(EcarosUtilities.java:484)

                            at de.procar.swing.SwingWorker$2.run(SwingWorker.java:121)

                            at java.lang.Thread.run(Unknown Source)

                         

                        this exception occurs only on windows clients and not at every invocation.

                        Linux clients are not affected.

                        On server side no exceptions are thrown.

                         

                        The result is, that the client is not usable anymore, which is not the main problem.

                        But on server side every failed invocation causes an Remoting task WAITING (on object monitor).

                        This blocks the Remoting Thread until the JBoss is restarted.

                        And the JBoss could only be stopped by killing the process:

                         

                        Here the thread dump of on of the blocked threads:

                         

                        "Remoting "future2012" task-3" prio=10 tid=0x00007f7398073800 nid=0xeca in Object.wait() [0x00007f73f3dfb000]

                           java.lang.Thread.State: WAITING (on object monitor)

                            at java.lang.Object.wait(Native Method)

                            - waiting on <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

                            at java.lang.Object.wait(Object.java:503)

                            at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:106)

                            at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:126)

                            at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:114)

                            at org.xnio.streams.BufferPipeOutputStream.flush(BufferPipeOutputStream.java:143)

                            - locked <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

                            at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:161)

                            - locked <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

                            at org.jboss.remoting3.remote.OutboundMessage.close(OutboundMessage.java:244)

                            - locked <0x00000000f8fb8298> (a org.xnio.streams.BufferPipeOutputStream)

                            at org.jboss.as.ejb3.remote.protocol.versionone.ChannelAssociation.releaseChannelMessageOutputStream(ChannelAssociation.java:85)

                            at org.jboss.as.ejb3.remote.protocol.AbstractMessageHandler.writeException(AbstractMessageHandler.java:118)

                            at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.processMessage(MethodInvocationMessageHandler.java:166)

                            at org.jboss.as.ejb3.remote.protocol.versionone.VersionOneProtocolChannelReceiver.handleMessage(VersionOneProtocolChannelReceiver.java:160)

                            at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:435)

                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                            at java.lang.Thread.run(Thread.java:744)

                         

                           Locked ownable synchronizers:

                            - <0x00000000e6512010> (a java.util.concurrent.ThreadPoolExecutor$Worker)

                         

                        We are running out of remoting task threads successivly.

                         

                        I think there is a bug within jboss remoting3.

                         

                        I keep on investigation and try to create an example.

                        This may take a few days.

                        • 9. Re: Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                          s.horn Newbie

                          Now i am able to reproduce the issue.

                          attached you can find the sourcecode for an example bean and client.

                           

                          The issue occures in combination JBoss 7.3.0 (Linux) and the ejb Client on an windows machine (Both running with JDK 1.7.0_51)

                          When running the client after a view successful invocations the following exception occurs

                          java.lang.IllegalStateException: EJBCLIENT000025: No EJB receiver available for handling [appName:, moduleName:remotingtest, distinctName:] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@19d136a
                                  at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:735)
                                  at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:116)
                                  at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:183)
                                  at org.jboss.ejb.client.EJBClientInvocationContext.retryRequest(EJBClientInvocationContext.java:208)
                                  at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:256)
                                  at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:265)
                                  at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:198)
                                  at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)
                                  at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)
                                  at com.sun.proxy.$Proxy0.sendMyContainer(Unknown Source)
                                  at de.procar.remotetest.RemotingTestClient.<init>(RemotingTestClient.java:48)
                                  at de.procar.remotetest.RemotingTestClient.main(RemotingTestClient.java:20)

                           

                          The clients invokes the method with an increasing Object in object count (See source code).

                          Suddently this kills the ejb receiver without any further exception than the no ejb receiver available exception.

                           

                          When the client is executed several times, Jboss stops cancelling the connections.

                          The number of Remoting Task Threads in State Waiting (on object monitor) increases until all tasks are blocked and Jboss is not responding on remoting requests anymore.

                           

                          Can anyone confirm that this is a Bug?

                          That would be very helpful.

                          A JBoss which is dying once a week because it is running out of remoting tasks is unusable.

                           

                          Thanks in advance.

                          • 10. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                            Sergey Kozhenkov Newbie

                            Hi,

                            I am wondering if you've been able to find a solution to that, I have similar issue here.

                             

                            Thank you.

                            • 12. Re: Help: java.lang.RuntimeException: Operation failed with status WAITING
                              s.horn Newbie

                              Hello,

                               

                              @Sergey Kozenkov: We have not found a solution, yet.

                              Currently we are switching to EAP 6.3 to see if the changes in marshalling and remoting will solve the problem.

                              Meanwhile we extended the remoting threads to 16 and restart the server every time when at least 3 threads are blocked. (Sometime 2 or 3 times the week )

                               

                              @ Philippe Marshall: We do not use EJB3 with SSL.

                              If the threads are blocked we analyse the threaddump and it never detects deadlocks. The issue must be somewhere else...