We would be glad to help you but I think this has to be taken case by case. If you provide detailed information about the problems perhaps we can help. I'm sure you get dig out the logs and give us some stacktraces.
I hoped I could get some general information about stability. Of course I have the log file, but it's 3.7GB big. It's not really comfortable getting some interesting data out of this file.
I think we will examine this behaviour later but for now we would like to have an idea what could be the problem.
BTW: How can I avoid having logged everything. There are a lot of DEBUG logs.
In my experience there is no silver-bullet solution on stability. You have to investigate the specific issues you are having and remediate. There is so many things at any layers that could cause instability.
First off you should review your logging solution to make your log file more manageable. Check out your conf/jboss-log4j.xml. You might want to add:
<category name="org.jboss.seam"> <priority value="INFO"/> </category> <category name="org.ajax4jsf"> <priority value="INFO"/> </category> <category name="org.springframework"> <priority value="INFO"/> </category> <category name="org.hibernate"> <priority value="INFO"/> </category> <category name="org.hibernate.type"> <priority value="TRACE" class="org.jboss.logging.XLevel"/> </category> <category name="org.hibernate.SQL"> <priority value="DEBUG"/> </category>
Generally you don't want to log lower than INFO unless you need all the info because you are debugging in a specific library. The hibernate DEBUG and TRACE settings above will get you all the SQL statements fired at DB along with parameter values.
if you are running under a *nix system you can use split command to split a big log file in chunks.
The other thing you can do is route specific log entries from java packages to different appenders so you don't have everything in the same file, however it does help to have everything in the same file to figure out what is going on across threads.
I'm not a log4j expert but it's probably possible to add the thread-name in each log entries. This would help you troubleshoot concurrency related issues.
If I recall correctly, I see a lot of
Could not find stateful bean: ...exceptions registered in the old JBoss Seam forum. Even I got some of them in my earlier times.
This is related to EJB Stateful Session Beans (SFSB) activation and passivation.
When Bean instance is not used for a while by client then EJB Container removes it from memory and puts it in secondary storage so that the memory can be reused. This is called Passivation.
When Client calls the bean instance again then Container takes the passivated bean from secondary storage and puts it in memory to serve the client request. This is called Activation.
The Stateful Session beans are not in a pasivated state for ever, they timeout (instance is completely removed from the secondary storage).
So what is happening seems to be the following:
- An SFSB is initialized within a conversational component. Suppose your conversation is configured to timeout in 5 minutes. Such a component get a reference to such SFSB.
- The SFSB is pasivated by the container.
- The SFSB timeout in 2 minute (a sample, this is dependant of the Application server defaults and configuration).
- Then the conversational component tries to access to the SFSB, but it does not exist anymore (it timed out), so you will get a nicest exception (Could not find stateful bean: ...).
That could be what is happening in your application.
Months ago I saw that Gavin King registered an issue in the JBoss EJB3 forum asking to them to make the default timeout of EJBs greater than the HTTP Session configured timeout, I don't remember the issue number and I don't know if it's resolved in the JBoss 5. Maybe you can look for it.
OK. We'll test again with modified log settings. Thank you for that hint.
As far as I know you can avoid such errors having the combination: SFSB timout (JBoss config) = Conversation timeout (Your app) = HttpSession timeout (Your app).
we are having the same problem with our live going Seam application. In the logs I get several messages like this:
|2008-10-13 18:04:34,993 WARN [org.jboss.seam.Component] Exception calling component @Destroy method: viewStateHolder javax.ejb.NoSuchEJBException: Could not find stateful bean: 61h5f1e-qnpm1u-fm6dzg6m-1-fm9aa9my-60l at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:390) at org.jboss.ejb3.cache.simple.SimpleStatefulCache.get(SimpleStatefulCache.java:375) at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:61) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateful.StatefulRemoveInterceptor.invoke(StatefulRemoveInterceptor.java:97) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77) at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:110) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateful.StatefulContainer.localInvoke(StatefulContainer.java:206) at org.jboss.ejb3.stateful.StatefulLocalProxy.invoke(StatefulLocalProxy.java:119)|
You can set the timeout settings
in the /server/default/deploy/ejb3-interceptors-aop.xml file of your JBoss installation like this:
<domain name="Stateful Bean" extends="Base Stateful Bean" inheritBindings="true"> <!-- NON Clustered cache configuration --> <annotation expr="!class(@org.jboss.annotation.ejb.cache.Cache) AND !class(@org.jboss.annotation.ejb.Clustered)"> @org.jboss.annotation.ejb.cache.Cache (org.jboss.ejb3.cache.simple.SimpleStatefulCache.class) </annotation> <annotation expr="!class(@org.jboss.annotation.ejb.cache.simple.PersistenceManager) AND !class(@org.jboss.annotation.ejb.Clustered)"> @org.jboss.annotation.ejb.cache.simple.PersistenceManager (org.jboss.ejb3.cache.simple.StatefulSessionFilePersistenceManager.class) </annotation> <annotation expr="!class(@org.jboss.annotation.ejb.cache.simple.CacheConfig) AND !class(@org.jboss.annotation.ejb.Clustered)"> @org.jboss.annotation.ejb.cache.simple.CacheConfig (maxSize=100000, idleTimeoutSeconds=7200, removalTimeoutSeconds=0) </annotation> ...
where idleTimeoutSeconds is the time in seconds after which the SFSB passivates and removalTimeoutSeconds is the timeout after which the complete SFSB is destroyed (0 means no timeout)(see here: https://www.jboss.org/community/docs/DOC-9628).
So in my config the timeout for passivation is 7200 seconds (2 hours). The problem is that I still get the error in my logs.
If my understanding is correct the SFSB is not destroyed after all (all it does is passivating after 2 hours!) because the removalTimeoutSeconds is set to infinite. So why do I get the errors assuming that my conversation is sill active? How can that be?
Any help is appreciated!
We are going to work on clustering for Seam 2.1.1 in tandem with Brain and co. from the JBoss Clustering team - so please, if you know of blogs about clustering Seam, or hints and tips for running in a cluster, or pain points, please add them to the knowledge base :-)
A few things:
- Don't use injections in multithreaded state (not safe) - use @Create or equiv
- Check logging thread locking - could be significant (very much so!) - use async append or shut down.
- Check 1000x setter calls problem - use lazy init
- Don't set the db isolation levels too high
- Don't use JMS bridging with JBoss MQ (duplicate messages can happen i have noticed), use clustering config (
allconfig) and MDB:s work like charm
- Make sure you handle timeouts gracefully (this you already noticed :))
When we discovered the thread locking problem, the app suddenly ran 10x as fast.
Alexander T wrote on Oct 14, 2008 20:50:
- Don't use injections in multithreaded state (not safe) - use @Create or equiv
Can you elaborate... What version of Seam?
Well, when injecting variables in application-scoped components I've had race conditions several times, with several contexts nulling and resetting those variables. But now when you mention it, that was mostly earlier on. Has there been a fix for that now? I believe I got that tip on the seam channel about a year ago or so. I'm doing initialization in @Create regularly now, so I haven't tried if it works lately...
Fixed in 2.1.0.CR1 I think (if we are talking about the same issue).