1 Reply Latest reply on Sep 8, 2015 2:49 PM by ivagulin

    Client application with remote EJB calls freezes.

    chrkoelle

      Hi

       

      We have a problem with remote EJB calls from one Wildfly (8.0.0.FINAL) instance to another. After an indetermined time the client application starts to be effectively non-responsive (we can see that there are some responses but only after some hundred seconds). A restart of only the client Wildfly instance solves the problem. A thread dump (jstack) of the client application JVM shows about 15 to 20 threads in state "WAITING (on object monitor)" with a stack traces that points to remote EJB calls (different EJBs and methods):

       

      "default task-14" prio=3 tid=0x000000010713b800 nid=0x1e9 in Object.wait() [0xfffffffe315f8000]

         java.lang.Thread.State: WAITING (on object monitor)

        at java.lang.Object.wait(Native Method)

        - waiting on <0xffffffff67d03020> (a java.lang.Object)

        at java.lang.Object.wait(Object.java:503)

        at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:397)

        - locked <0xffffffff67d03020> (a java.lang.Object)

        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:202)

        at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)

        at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)

        at com.sun.proxy.$Proxy173.getNews(Unknown Source)

       

      During normal operation these waiting thread are not present (or only one or two).

       

      The configuration of the remote ejb calls has been done according to the description in the documentation:

       

      https://docs.jboss.org/author/display/WFLY8/EJB+invocations+from+a+remote+server+instance

       

      with only little changes. Here are the relevant sections of the configuration files:

       

      standalone.xml:

       

              <subsystem xmlns="urn:jboss:domain:ejb3:2.0">

                ...

                <remote connector-ref="http-remoting-connector" thread-pool-name="ejb-remote"/>

                <thread-pools>

                    <thread-pool name="default">

                        <max-threads count="50"/>

                        <keepalive-time time="100" unit="milliseconds"/>

                    </thread-pool>

                    <thread-pool name="ejb-remote">

                        <max-threads count="50"/>

                        <keepalive-time time="100" unit="milliseconds"/>

                    </thread-pool>

                </thread-pools>

                ...

              </subsystem>

       

              <subsystem xmlns="urn:jboss:domain:remoting:2.0">

                <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>           

                <outbound-connections>

                  <remote-outbound-connection name="remote-ejb-connection" outbound-socket-binding-ref="remote-ejb-binding" protocol="http-remoting" security-realm="ejb-security-realm" username="ejb">

                    <properties>

                      <property name="SASL_POLICY_NOANONYMOUS" value="false"/>

                      <property name="SSL_ENABLED" value="false"/>

                      <property name="KEEP_ALIVE" value="true"/>

                      <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="1000"/>

                    </properties>

                  </remote-outbound-connection>

                </outbound-connections>

              </subsystem>

       

            <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

              ...

              <outbound-socket-binding name="remote-ejb-binding">

                <remote-destination host="server.remote.com" port="8080"/>

              </outbound-socket-binding>                   

            </socket-binding-group>

       

      The problems seems to be related to the system load but i was unable to reproduce this in the development environment. We ran the application in the JBoss 7.1 server before without any such problems.

       

      I searched for any hints for this problem in the forums, in JIRA and the rest of the web without any success. Does anybody have any idea?

       

      Christian

        • 1. Re: Client application with remote EJB calls freezes.
          ivagulin

          was able to solve it by increasing max-task-threads for io subsystem

           

              <subsystem xmlns="urn:jboss:domain:io:1.1">
                  <worker name="default" task-max-threads="150"/>
                  <buffer-pool name="default"/>
              </subsystem>

           

           

          Looks like every remote ejb call requires spare task-thread to proceed. If you make ejb-remote call from servlet. 2 threads required to accomplish http request: 1 for servlet thread, another to accept ejb thread response