WildFly 9 EJB Client/Server Configuration Advice

[jfisherdev]

My organization is currently migrating an application from JBoss AS 4.2.2 to WildFly 9.0.2.final.

We need some guidance in the area of EJB client and server configuration, as the default configuration that ships with WildFly is most likely unsuitable for our purposes.

I will describe the application and deployment configuration, and then identify the three areas where we need some guidance.

The application is an EAR application deployed to a standalone WildFly 9 server. Most interactions with the server side are remote EJB invocations on stateless session beans from a standalone Swing GUI client application. Some areas of the client application make use of SwingWorker objects to make remote EJB invocations in the background/concurrently to keep the client responsive. The JBoss EJB Client API with scoped contexts is being used to make these remote calls.

The application deploys two stateless session beans--one for container managed transactions and one for bean managed transactions--that receive invocation metadata and use that to reflectively invoke the business object targeted by the client. This keeps the number of EJB components small, though it does mean the target business objects that would normally be deployed as stateless session bean components do not have the benefits of being a true EJB component; however, this has not been an issue for us.

The minimum number of clients being served is around 300, the maximum is around 1000, and the average is about 650.

The three areas I believe we need some guidance are as follows:

1. EJB3/Remoting subsystem configuration. From what I have read, these seem to be closely related. There are some attributes that seem important for tuning, but I am not sure how to determine what these should be.

• For EJB3, the max threads attribute of the thread pool  as well as the max pool size in the slsb-strict-max-pool.
• In the Remoting subsystem, the number of max inbound/outbound channels and messages.

2. EJB client configuration. I have looked over the client properties documentation https://docs.jboss.org/author/display/EJBCLIENT/Overview+of+Client+properties

but there are some properties I am not sure how to configure that seem like they would be important. In particular, the MAX_INBOUND_MESSAGES and MAX_OUTBOUND_MESSAGES as well as the various timeout properties.

3. IO subsystem and general considerations for NIO/XNIO. NIO/XNIO worker and buffer pool configuration seems to be important, as the EJB3, Remoting, and Undertow subsystems ultimately appear to be tied to the worker(s) configured in the IO subsystem. As far as I know, NIO was not a consideration that needed to be made in JBoss AS 4.2.2, and it is fairly new to me, though I am learning more about it. I am curious if any considerations for NIO need to be made on the application side.

Any guidance or information on this subject would be very much appreciated.

[jfisherdev]

I would like to add that we tried switching the client to make some of these simultaneous background calls via web service calls rather than the EJB client API and that this seemed to perform better. We are not planning to switch to this approach and would like to use the EJB client API; however, our findings seem to suggest there is some configuration we need to do.

[jfisherdev]

We have done some monitoring of the ejb3 subsystem via JMX. One thing we noticed is that the Current thread count attribute for the thread pool always seems to start at 0 and continues to grow until it reaches the Max threads. The number is strictly increasing. Is it ever supposed to decrease?

Also, the current number of max threads we have set is 500. Does anyone know if this number suitable based on the requirements I presented in my original post?

We are still having some issues with client configuration. When attempting to make simultaneous EJB client calls across background threads, the calls seem to be queued up and made sequentially, rather than in parallel. As I mentioned in my previous post, we were able to get remote calls to be processed in parallel when using web service calls rather than EJB client calls, but this is not something we want to do and seems to be a configuration issue.

[jfisherdev]

We discovered a few client configuration and implementation issues that once fixed improved performance significantly.

The main issues in this case were:

• The client was creating and closing a new scoped EJB client context for every remote EJB invocation. This was the main cause of our performance issues. Now the client creates and caches a scoped EJB client context for each unique EJB client configuration or server and will not close them until the client is exiting. I think part of the issue was when I looked at the examples about scoped EJB client contexts and the importance of closing them I misinterpreted some of this information when implementing the client. In particular I took the "close the context when it is no longer needed" direction to mean after every invocation.
• We had the jboss.remoting.debug-buffer-leaks property set to true on both the client and the server. Removing this property improved performance also. We do have the jboss.remoting.pooled-buffers property set to false, as we were running into the known memory leak issues when it was not set.

Overall performance has been good with WildFly once we made these changes.

If anyone has any information or guidance to share about tuning EJB configuration on either the client or server side we would still appreciate it.

[jaikiran]

jfisherdev wrote:

We discovered a few client configuration and implementation issues that once fixed improved performance significantly.

The main issues in this case were:

• The client was creating and closing a new scoped EJB client context for every remote EJB invocation. This was the main cause of our performance issues. Now the client creates and caches a scoped EJB client context for each unique EJB client configuration or server and will not close them until the client is exiting. I think part of the issue was when I looked at the examples about scoped EJB client contexts and the importance of closing them I misinterpreted some of this information when implementing the client. In particular I took the "close the context when it is no longer needed" direction to mean after every invocation.

To add some details to this - The EJB client context will be backed by one or more "EJB receivers" (WildFly specific terminology) and the remote variants of these receivers actually are backed by remote connections. So if the client context is opened and closed every invocation, it will lead to connection creation and closing every invocation, which isn't a good thing. The client context can be kept open as long as it makes sense for the application use case, but as noted it needs to be closed at the right time so that it doesn't leak the connections - just like JDBC connection handling.