Client application with remote EJB calls freezes.
chrkoelle Jun 30, 2014 6:45 AMHi
We have a problem with remote EJB calls from one Wildfly (8.0.0.FINAL) instance to another. After an indetermined time the client application starts to be effectively non-responsive (we can see that there are some responses but only after some hundred seconds). A restart of only the client Wildfly instance solves the problem. A thread dump (jstack) of the client application JVM shows about 15 to 20 threads in state "WAITING (on object monitor)" with a stack traces that points to remote EJB calls (different EJBs and methods):
"default task-14" prio=3 tid=0x000000010713b800 nid=0x1e9 in Object.wait() [0xfffffffe315f8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xffffffff67d03020> (a java.lang.Object)
at java.lang.Object.wait(Object.java:503)
at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:397)
- locked <0xffffffff67d03020> (a java.lang.Object)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:202)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:181)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:144)
at com.sun.proxy.$Proxy173.getNews(Unknown Source)
During normal operation these waiting thread are not present (or only one or two).
The configuration of the remote ejb calls has been done according to the description in the documentation:
https://docs.jboss.org/author/display/WFLY8/EJB+invocations+from+a+remote+server+instance
with only little changes. Here are the relevant sections of the configuration files:
standalone.xml:
<subsystem xmlns="urn:jboss:domain:ejb3:2.0">
...
<remote connector-ref="http-remoting-connector" thread-pool-name="ejb-remote"/>
<thread-pools>
<thread-pool name="default">
<max-threads count="50"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
<thread-pool name="ejb-remote">
<max-threads count="50"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
...
</subsystem>
<subsystem xmlns="urn:jboss:domain:remoting:2.0">
<http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
<outbound-connections>
<remote-outbound-connection name="remote-ejb-connection" outbound-socket-binding-ref="remote-ejb-binding" protocol="http-remoting" security-realm="ejb-security-realm" username="ejb">
<properties>
<property name="SASL_POLICY_NOANONYMOUS" value="false"/>
<property name="SSL_ENABLED" value="false"/>
<property name="KEEP_ALIVE" value="true"/>
<property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="1000"/>
</properties>
</remote-outbound-connection>
</outbound-connections>
</subsystem>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
...
<outbound-socket-binding name="remote-ejb-binding">
<remote-destination host="server.remote.com" port="8080"/>
</outbound-socket-binding>
</socket-binding-group>
The problems seems to be related to the system load but i was unable to reproduce this in the development environment. We ran the application in the JBoss 7.1 server before without any such problems.
I searched for any hints for this problem in the forums, in JIRA and the rest of the web without any success. Does anybody have any idea?
Christian