[brian.stansberry]

I see it. You are using :

import org.jboss.annotation.ejb.Clustered;

which was removed a long time ago as part of the reorganization of EJB3 annotations; now it is:

import org.jboss.ejb3.annotation.Clustered;

[andy.miller]

I see it. You are using :

import org.jboss.annotation.ejb.Clustered;

which was removed a long time ago as part of the reorganization of EJB3 annotations; now it is:

import org.jboss.ejb3.annotation.Clustered;

Okay, so I have to rebuild this against the AS 5 CR 2 libraries then. I was hoping to just be able to take what I had built before, but that's okay. I can add the AS 5 CR 2 libraries to my project manually, and rebuild.

[alrubinger]

I had to work on FY'10 budget stuff until late last night, and I had to sit on a call from 6:30 this morning to go over it with a bunch of folks, so I'm just now getting around to creating a slimmed down test case.

Wasn't meant to pressure you, just showing what I'll need to reflect this in the community suite. :)

S,
ALR

[wolfc]

That takes care of the cache configuration, but I also want to know what's behind this passivation issue.

Is passivating or activating the culprit?
On what OS is this?

If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.

[brian.stansberry]

What I love about this Andy is it appears you are demonstrating taht what I expect should work does work -- a bean annotated @Clustered w/o a remote interface:

https://jira.jboss.org/jira/browse/EJBTHREE-1540


Andrew, remember or chat about this a couple months back?

[brian.stansberry]

"wolfc" wrote:
That takes care of the cache configuration, but I also want to know what's behind this passivation issue.

Is passivating or activating the culprit?
On what OS is this?

If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.

Needs to be proven of course, but SimpleStatefulCache.SessionTimeoutTask holds a lock on the cacheMap while it iterates through the contents and passivates. If there are a lot of beans (knowing Andy, this is a load test and there are) that could take a while. Any SimpleStatefulCache.get() request will block in the meantime.

[alrubinger]

"wolfc" wrote:
If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.

There's no contention issue. Other Threads will block.

Bet you a Bacardi 'n Coke.

S,
ALR

[andy.miller]

"wolfc" wrote:
That takes care of the cache configuration, but I also want to know what's behind this passivation issue.

Is passivating or activating the culprit?
On what OS is this?

If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.

The issue is clearly passivation. The passivation count is zero on the first occurrence, and everything hangs, and then the passivation count is non-zero. The only time the hang occurs is when passivation happens. My typical response times on calling the stateful session bean and returning with the result set are between 7 and 40ms.

This is being run on Fedora 9, with our OpenJDK 6 JVM.

After rebuilding with the AS 5 CR 2 libraries, the problem does go away, because I now have confirmed that I'm using the StatefulTreeCache, which is the default in the ejb3-interceptors-aop.xml based on the fact that I only have the @Clustered annotation:

<!-- Clustered cache configuration -->
 <annotation expr="!class(@org.jboss.ejb3.annotation.Cache) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.Cache ("StatefulTreeCache")
 </annotation>
 <annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
 </annotation>

Clearly the synchronization in the SimpleStatefulCache is a real problem. If I weren't running in a clustered configuration (and didn't have the issue I had because I didn't rebuild with the AS 5 libraries), I would still have the same problem.

[andy.miller]

"ALRubinger" wrote:

"wolfc" wrote:
If it was the sync on cacheMap I would say a deadlock would happen, not some timeout of 30 seconds.

There's no contention issue. Other Threads will block.

Bet you a Bacardi 'n Coke.

S,
ALR

It's not really a timeout. Its just everything waiting for the passivation to complete. Once it completes, everything starts running again. When I get the test case, I think you will see what I mean. I believe the code you already identified is the real culprit.

[andy.miller]

I see it. You are using :

import org.jboss.annotation.ejb.Clustered;

which was removed a long time ago as part of the reorganization of EJB3 annotations; now it is:

import org.jboss.ejb3.annotation.Clustered;

Okay, I rebuilt the application with the AS 5 libraries, and I am no longer getting the SimpleStatefulCache, but instead am getting the StatefulTreeCache.

You cannot configure the StatefulTreeCache (at least I can't see how), so my throughput has now dropped, even though I don't have the hang situation anymore. The StatefulTreeCache doesn't appear to have a maxSize, and it seems to be limited to 10,000 beans.

So, the question I have, is how do I get it to pick up the EJB3SFSBSClusteredCache. It appears I would need to add an annotation (@Cache or @CacheConfig), but the documentation for both of these is out-of-date, and the source has no explanation (no javadoc).

Any ideas?

[brian.stansberry]

You would add an @CacheConfig annotation:

@org.jboss.ejb3.annotation.CacheConfig (name="sfsb-cache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)

The maxSize param controls the max number of beans.

If you don't configure that, and your bean is @Clustered, per the deploy/ejb3-interceptors-aop.xml I have in my workspace, the following should be added to your bean:

@org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)

It already has a maxSize of 100,000 so I'm not sure how you're ending up with only 10,000. I'll poke around.

The name="jboss.cache:service=EJB3SFSBClusteredCache" bit needs to be fixed: https://jira.jboss.org/jira/browse/JBAS-6128 . The way StatefulTreeCache uses that value in AS 5 is to drive a lookup in the CacheManager service, deployed via deploy/cluster/jboss-cache-manager.sar. CacheManager is an on-demand factory for JBoss Cache instances. It's maintains a registry of named JBC configurations, and provides caches based on the name.

The name here should be "sfsb-cache". The cache manager is configured to understand the old 4.x name "jboss.cache:service=EJB3SFSBClusteredCache" to be an alias for "sfsb-cache". That allows pre-existing code to work.

If you actually wanted to play around with the "sfsb-cache" JBC configuration, you could edit deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-cache-configs.xml. But I don't think that's what you are looking for; you don't control things like the max # of beans via JBC configuration; it's via @CacheConfig.

Obviously this needs documenting. :(

Does the deploy/ejb3-interceptors-aop.xml file on your JBoss install include this?

<annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
 </annotation>

If your maxSize value were something different, that would explain it.

[brian.stansberry]

I just moved the above mentioned JIRA to ejb3: https://jira.jboss.org/jira/browse/EJBTHREE-1541

[andy.miller]

You would add an @CacheConfig annotation:

@org.jboss.ejb3.annotation.CacheConfig (name="sfsb-cache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)

The maxSize param controls the max number of beans.

If you don't configure that, and your bean is @Clustered, per the deploy/ejb3-interceptors-aop.xml I have in my workspace, the following should be added to your bean:

@org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)

It already has a maxSize of 100,000 so I'm not sure how you're ending up with only 10,000. I'll poke around.

The name="jboss.cache:service=EJB3SFSBClusteredCache" bit needs to be fixed: https://jira.jboss.org/jira/browse/JBAS-6128 . The way StatefulTreeCache uses that value in AS 5 is to drive a lookup in the CacheManager service, deployed via deploy/cluster/jboss-cache-manager.sar. CacheManager is an on-demand factory for JBoss Cache instances. It's maintains a registry of named JBC configurations, and provides caches based on the name.

The name here should be "sfsb-cache". The cache manager is configured to understand the old 4.x name "jboss.cache:service=EJB3SFSBClusteredCache" to be an alias for "sfsb-cache". That allows pre-existing code to work.

If you actually wanted to play around with the "sfsb-cache" JBC configuration, you could edit deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-cache-configs.xml. But I don't think that's what you are looking for; you don't control things like the max # of beans via JBC configuration; it's via @CacheConfig.

Obviously this needs documenting. :(

Does the deploy/ejb3-interceptors-aop.xml file on your JBoss install include this?

<annotation expr="!class(@org.jboss.ejb3.annotation.CacheConfig) AND class(@org.jboss.ejb3.annotation.Clustered)">
 @org.jboss.ejb3.annotation.CacheConfig (name="jboss.cache:service=EJB3SFSBClusteredCache", maxSize=100000, idleTimeoutSeconds=300, removalTimeoutSeconds=0)
 </annotation>

If your maxSize value were something different, that would explain it.

I do have the above in my ejb3-interceptors-aop.xml.

Here is the behavior I see:

With just @Clustered and no Cache or CacheConfig annotation, I get the StatefulTreeCache.

I think you say that using jboss.cache:service=EJB3SFSBClusteredCache doesn't work, and you would be correct. I had that, and I get no deployment error, but I still get StatefulTreeCache.

With @Clustered and @CacheConfig(name="sfsb-cache", maxSize=125000) I still get StatefulTreeCache (not the EJB3SFSBStatefulTreeCache), and the size shows -1 in the jmx-console, and when I run it I still only get 10,000 as the max size. I just figured 10,000 because it never goes above that when refreshing, and when it gets close to that size is when passivation occurs.

This just doesn't appear to be working correctly at all.

[brian.stansberry]

I do have the above in my ejb3-interceptors-aop.xml.

Here is the behavior I see:

With just @Clustered and no Cache or CacheConfig annotation, I get the StatefulTreeCache.

Good.

I think you say that using jboss.cache:service=EJB3SFSBClusteredCache doesn't work, and you would be correct. I had that, and I get no deployment error, but I still get StatefulTreeCache.

No, that's not what I meant; what you see is what I expect.

There's a couple of concepts here; one of which users normally don't need to worry about, the other they do.

The first is what implementation of the "StatefulCache" interface EJB3 will use. This is the one users normally don't have to worry about. That interface defines the SPI the EJB3 container uses to interact with the caching subsystem. "StatefulTreeCache" is an implementation of that SPI that knows how to interact with JBC. "SimpleStatefulCache" is an implementation that works with the filesystem.

You only became aware of this stuff because you had the @Clustered package issue that caused EJB3 to use the wrong one. With that fixed, with @Clustered you should always see "StatefulTreeCache".

The second concept is how to configure whatever impl of StatefulCache is used. That's an end user concern and is done via the @CacheConfig annotation. The ejb3-interceptors-aop.xml stuff you saw is just a way of setting defaults if you don't add the annotation yourself.

StatefulTreeCache uses the @CacheConfig.name attribute to tell it what JBoss Cache configuration to use. It gets the JBC instance from the CacheManager service by passing in the @CacheConfig.name value. The standard value it should pass in is "sfsb-cache", but if it passes in the legacy AS 4.x default value of "jboss.cache:service=EJB3SFSBClusteredCache", the CacheManager knows that's just an alias for the AS 5.x default of "sfsb-cache".

So, this part sounds like its working just fine.

With @Clustered and @CacheConfig(name="sfsb-cache", maxSize=125000) I still get StatefulTreeCache (not the EJB3SFSBStatefulTreeCache), and the size shows -1 in the jmx-console, and when I run it I still only get 10,000 as the max size. I just figured 10,000 because it never goes above that when refreshing, and when it gets close to that size is when passivation occurs.

Yes, why the config value of maxSize=100,000 doesn't seem to be respected is what I need to figure out.

[andy.miller]

I do have the above in my ejb3-interceptors-aop.xml.

Here is the behavior I see:

With just @Clustered and no Cache or CacheConfig annotation, I get the StatefulTreeCache.

Good.

I think you say that using jboss.cache:service=EJB3SFSBClusteredCache doesn't work, and you would be correct. I had that, and I get no deployment error, but I still get StatefulTreeCache.

No, that's not what I meant; what you see is what I expect.

There's a couple of concepts here; one of which users normally don't need to worry about, the other they do.

The first is what implementation of the "StatefulCache" interface EJB3 will use. This is the one users normally don't have to worry about. That interface defines the SPI the EJB3 container uses to interact with the caching subsystem. "StatefulTreeCache" is an implementation of that SPI that knows how to interact with JBC. "SimpleStatefulCache" is an implementation that works with the filesystem.

You only became aware of this stuff because you had the @Clustered package issue that caused EJB3 to use the wrong one. With that fixed, with @Clustered you should always see "StatefulTreeCache".

The second concept is how to configure whatever impl of StatefulCache is used. That's an end user concern and is done via the @CacheConfig annotation. The ejb3-interceptors-aop.xml stuff you saw is just a way of setting defaults if you don't add the annotation yourself.

StatefulTreeCache uses the @CacheConfig.name attribute to tell it what JBoss Cache configuration to use. It gets the JBC instance from the CacheManager service by passing in the @CacheConfig.name value. The standard value it should pass in is "sfsb-cache", but if it passes in the legacy AS 4.x default value of "jboss.cache:service=EJB3SFSBClusteredCache", the CacheManager knows that's just an alias for the AS 5.x default of "sfsb-cache".

So, this part sounds like its working just fine.

With @Clustered and @CacheConfig(name="sfsb-cache", maxSize=125000) I still get StatefulTreeCache (not the EJB3SFSBStatefulTreeCache), and the size shows -1 in the jmx-console, and when I run it I still only get 10,000 as the max size. I just figured 10,000 because it never goes above that when refreshing, and when it gets close to that size is when passivation occurs.

Yes, why the config value of maxSize=100,000 doesn't seem to be respected is what I need to figure out.

Thanks for explaining. I was misunderstanding the legacy part.

One final question. If I remove the @CacheConfig annotation, can I add the maxSize and the other parameters to the ejb3-interceptors-aop.xml, like it has for the other two entries?