Passivating EJB 3 Stateful Session Beans stops the World!

[andy.miller]

Carlo and Andrew,

I have been running the same load test that I ran for my EAP 4.x performance tuning presentation (uses the EJB 3 Stress Test application we use in QE), and I have discovered what appears to be a pretty major problem. When I first kick-off a run, everything is running quite smoothly and I am actually getting slightly higher results than with EAP 4.2. Then everything just stops. The test just seems to be hung for anywhere between 30 seconds and a minute. Then everything starts again, and it runs like it was running before. In trying to correlate what was happening I found something quite interesting.

In looking at the JMX console for my stateful session bean, I can see that when everything stops, is when the container is passivating bean instances. In fact, when everything stops, that is the only time that the passivation counts increments in the JMX console. On EAP 4.2/4.3 I noticed similar behavior, but never correlated it to anything, because it would stop only for 5 to 10 seconds, never this extended period of time. Even with these hangs, we are only about 8% slower than before, and if these hangs were eliminated, we would be faster by about 3%. Of course, I am using OpenJDK 6, versus using the Sun JDK 5, and the improvement may be just because of the JVM, but even if that is the case, we are probably a wash in terms of performance.

In any case, this is a serious issue that needs investigation. Here is some more relevant information:

The default cache configuration is being used, which should be the StatefulTreeCache, since I have the @Clustered annotation on my stateful bean.

I can probably reduce the code down to just the servlet and stateful session bean, and strip out the database access, and just put a sleep in the method implementation.

[andy.miller]

Some additional information:

I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache. I'm not really sure why. Perhaps something I need to have on the @Clustered annotation?

Right now, even though I am using a clustered configuration of AS 5 CR2, I don't have another node joining the cluster, but I wouldn't expect the cache to be different based on that.

[brian.stansberry]

I'll have a look at how the choice of cache implementation is working in the absence of an explicit @CacheConfig annotation. I'm assuming there's no @CacheConfig annotation; please correct me if I'm wrong.

[alrubinger]

I can probably reduce the code down to just the servlet and stateful session bean, and strip out the database access, and just put a sleep in the method implementation.

In the end we're going to need a more isolated case that may involve invocations on an SFSB alone, or preferably something to test Cache passivation directly. Then we may apply the same test to any of our cache implementations.

S,
ALR

[alrubinger]

I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache. I'm not really sure why. Perhaps something I need to have on the @Clustered annotation?

This shouldn't be. We add the proper default annotations if not specified in AOP (ejb3-interceptors-aop.xml):

<!-- Clustered cache configuration -->
 <annotation expr="!class(@org.jboss.ejb3.annotation.Cache) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache")
 </annotation>
 <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
 </annotation>

S,
ALR

[alrubinger]

I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache.

Something I noted w/ SimpleStatefulCache:

Without seeing this firsthand, my educated guess is in SessionTimeoutTask.run() where we synchronize on the entire SimpleStatefulCache.cacheMap to do passivation; this will queue up all other Threads looking to create or get a session as they wait to acquire the lock.

http://anonsvn.jboss.org/repos/jbossas/projects/ejb3/trunk/core/src/main/java/org/jboss/ejb3/cache/simple/SimpleStatefulCache.java

S,
ALR

[andy.miller]

I'll have a look at how the choice of cache implementation is working in the absence of an explicit @CacheConfig annotation. I'm assuming there's no @CacheConfig annotation; please correct me if I'm wrong.

Yes, you are correct, there is not @CacheConfig annotation. Only the @Clustered annotation.

[andy.miller]

"ALRubinger" wrote:

I tested changing the cache maxSize parameter with ejb3-interceptors-aop.xml, and discovered that I am actually using the SimpleStatefulCache. I'm not really sure why. Perhaps something I need to have on the @Clustered annotation?

This shouldn't be. We add the proper default annotations if not specified in AOP (ejb3-interceptors-aop.xml):

<!-- Clustered cache configuration -->
 <annotation expr="!class(@org.jboss.ejb3.annotation.Cache) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache")
 </annotation>
 <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
 </annotation>

S,
ALR

I do have what you list above in my configuration. Just before it is also the non-clustered configuration, and that is what I'm picking up, based on my experimentation.

I think Brian will discover a problem with the way this is being selected when there is no @CacheConfig annotation.

[andy.miller]

"ALRubinger" wrote:

I can probably reduce the code down to just the servlet and stateful session bean, and strip out the database access, and just put a sleep in the method implementation.

In the end we're going to need a more isolated case that may involve invocations on an SFSB alone, or preferably something to test Cache passivation directly. Then we may apply the same test to any of our cache implementations.

S,
ALR

I will work on cutting out just the code for the Stateful Session Bean, so it can be turned into an isolated test case tomorrow.

[brian.stansberry]

Andy,

Can you post the class declaration for the troublesome bean (along w/ any class annotations) plus any ejb-jar.xml or jboss.xml that impacts the bean?

Looking at this, how it determines the cache implementation to use looks pretty straightforward. I misspoke before; it's driven by the org.jboss.ejb3.annotation.Cache annotation, not @CacheConfig.

[andy.miller]

Brian,

Sure thing. I don't have any ejb-jar.xml or jboss.xml files at all. I tried to use all the defaults as well (convention over configuration) for everything that I could. So here is the class declaration:

package services.ejb;

import java.io.Serializable;
import java.util.ArrayList;
import java.util.List;

import javax.ejb.Remove;
import javax.ejb.Stateful;
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;

import org.jboss.annotation.ejb.Clustered;
import services.entities.Order;

@Clustered
@Stateful
public class OrderInquiryBean implements OrderInquiry, Serializable {

Here is the interface as well:

package services.ejb;

import java.util.List;
import services.entities.Order;

public interface OrderInquiry {

 public int getPaginationForInquiries();

 public List<Order> findOrdersByCustomer(long customerId);

 public void cancelSession();

}

Like I said. I kept everything as simple as possible.

[andy.miller]

Andy,

I misspoke before; it's driven by the org.jboss.ejb3.annotation.Cache annotation, not @CacheConfig.

Actually, you were correct according to the documentation. The @Cache annotation is for caching entities, and the @CacheConfig is for StatefulSessionBeans, like what I'm doing.

[brian.stansberry]

I believe you are thinking of org.hibernate.annotations.Cache, which is used for entities. The org.jboss.ejb3.annotation.Cache annotation is new and is used to drive what SFSB cache implementation to use. As Andrew's snippet shows, the presence of @Clustered should drive the addition of @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache") which in turn should result in StatefulTreeCache being used.

But apparently it doesn't. :)

Thanks for the config details; that's what I guessed you were doing, but now I know. Hopefully if I can reproduce.

[alrubinger]

Andy:

And the test client?

S,
ALR

[andy.miller]

"ALRubinger" wrote:
Andy:

And the test client?

S,
ALR

I had to work on FY'10 budget stuff until late last night, and I had to sit on a call from 6:30 this morning to go over it with a bunch of folks, so I'm just now getting around to creating a slimmed down test case.