Richfaces + Seam --> CPU100%

[pramod_bs]

Richfaces -- 3.3.1 GA
Seam --> 2.1.1
JBoss 4.2.1

I have 100% CPU utilization. JMX-Console dump shows this as culprit. I might be doing somerthing wrong but I am not sure what could cause this issue. Please help. My application is in PRODUCTION...

Thread: http-159.142.4.58-8180-13 : priority:5, demon:true, threadId:87, threadState:RUNNABLE, lockName:null
 java.util.HashMap.get(HashMap.java:346)
 org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:965)
 org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:984)
 org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:943)
 org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:361)
 org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:82)
 org.richfaces.model.ModifiableModel.walk(ModifiableModel.java:156)
 org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1133)
 org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:104)
 org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:88)
 org.richfaces.renderkit.AbstractTableRenderer.encodeTBody(AbstractTableRenderer.java:90)
 org.richfaces.renderkit.AbstractTableRenderer.encodeChildren(AbstractTableRenderer.java:97)
 javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
 org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:277)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:166)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:83)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:157)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:83)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:157)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:83)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:157)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:83)
 org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:157)
 org.ajax4jsf.renderkit.AjaxContainerRenderer.encodeAjax(AjaxContainerRenderer.java:123)
 org.ajax4jsf.component.AjaxViewRoot.encodeAjax(AjaxViewRoot.java:677)
 org.ajax4jsf.component.AjaxViewRoot.encodeChildren(AjaxViewRoot.java:548)
 javax.faces.component.UIComponent.encodeAll(UIComponent.java:886)
 com.sun.facelets.FaceletViewHandler.renderView(FaceletViewHandler.java:592)
 org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100)
 org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176)
 com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
 com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
 com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
 javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
 org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
 org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
 org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
 org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
 org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178)
 org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
 org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:368)
 org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:495)
 org.jboss.seam.web.Ajax4jsfFilter.doFilter(Ajax4jsfFilter.java:56)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
 org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:58)
 org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
 org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
 org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
 org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
 org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
 org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
 org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
 org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:543)
 org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
 org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
 org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
 org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 java.lang.Thread.run(Thread.java:595)

[pramod_bs]

Please help.

[gonzalad]

Hi,

We need more info I think.
What is the offending page ?
What's the content of your web.xml ?

[pramod_bs]

The problem is we don't know which page is causing the issue. There is consistent traffic to our website. At somepoint the error is occuring. The thread dump is what is given in my original post.
web.xml is below.
CPU consumption below. All the following threads have the above stacktrace

Thread Name CPU (milliseconds)
http-159.142.4.58-8180-13 44084484
http-159.142.4.58-8180-19 44082265
http-159.142.4.58-8180-57 42665000
http-159.142.4.58-8180-6 42598093
http-159.142.4.58-8180-27 42547218
http-159.142.4.58-8180-78 42491109
http-159.142.4.58-8180-73 42454640
ScannerThread 314953

It seems like this is the code may b going into some infinite loop.

 java.util.HashMap.get(HashMap.java:346)
 org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:965)

web.xml

<?xml version="1.0" ?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
 version="2.5">

 <!-- RichFaces --><!--
 <context-param>
 <param-name>org.richfaces.SKIN</param-name>
 <param-value>#{themeSelector.theme}</param-value>
 </context-param>

--><context-param>
 <param-name>org.richfaces.SKIN</param-name>
 <param-value>blueSky</param-value>
</context-param>

 <!-- Suppress spurious stylesheets -->

 <context-param>
 <param-name>org.richfaces.CONTROL_SKINNING</param-name>
 <param-value>disable</param-value>
 </context-param>

 <context-param>
 <param-name>org.richfaces.CONTROL_SKINNING_CLASSES</param-name>
 <param-value>disable</param-value>
 </context-param>
<!--
 <context-param>
 <param-name>org.ajax4jsf.xmlparser.ORDER</param-name>
 <param-value>NONE</param-value>
 </context-param>
 -->
 <context-param>
 <param-name>javax.faces.STATE_SAVING_METHOD</param-name>
 <param-value>server</param-value>
 </context-param>

 <context-param>
 <param-name>com.sun.faces.responseBufferSize</param-name>
 <param-value>500000</param-value>
 </context-param>

 <context-param>
 <param-name>facelets.BUFFER_SIZE</param-name>
 <param-value>500000</param-value>
 </context-param>

 <context-param>
 <param-name>facelets.REFRESH_PERIOD</param-name>
 <param-value>-1</param-value>
 </context-param>

 <!--
 Change load strategy to DEFAULT to disable sending scripts/styles as
 packs
 -->

 <context-param>
 <param-name>org.richfaces.LoadStyleStrategy</param-name>
 <param-value>ALL</param-value>
 </context-param>

 <context-param>
 <param-name>org.richfaces.LoadScriptStrategy</param-name>
 <param-value>ALL</param-value>
 </context-param>

 <context-param>
 <param-name>org.ajax4jsf.COMPRESS_SCRIPT</param-name>
 <param-value>false</param-value>
 </context-param>

 <context-param>
 <param-name>org.ajax4jsf.COMPRESS_STYLE</param-name>
 <param-value>false</param-value>
 </context-param>
 <!--
 <context-param> <param-name>org.ajax4jsf.VIEW_HANDLERS</param-name>
 <param-value>com.sun.facelets.FaceletViewHandler</param-value>
 </context-param>
 -->
 <!-- Seam -->

 <listener>
 <listener-class>org.jboss.seam.servlet.SeamListener</listener-class>
 </listener>

 <filter>
 <filter-name>Seam Filter</filter-name>
 <filter-class>org.jboss.seam.servlet.SeamFilter</filter-class>
 </filter>

 <filter-mapping>
 <filter-name>Seam Filter</filter-name>
 <url-pattern>/*</url-pattern>
 </filter-mapping>

 <servlet>
 <servlet-name>Seam Resource Servlet</servlet-name>
 <servlet-class>org.jboss.seam.servlet.SeamResourceServlet</servlet-class>
 </servlet>

 <servlet-mapping>
 <servlet-name>Seam Resource Servlet</servlet-name>
 <url-pattern>/seam/resource/*</url-pattern>
 </servlet-mapping>

 <!-- Facelets development mode (disable in production) -->

 <context-param>
 <param-name>facelets.DEVELOPMENT</param-name>
 <param-value>false</param-value>
 </context-param>

 <!-- JSF -->

 <context-param>
 <param-name>javax.faces.DEFAULT_SUFFIX</param-name>
 <param-value>.xhtml</param-value>
 </context-param>

 <servlet>
 <servlet-name>Faces Servlet</servlet-name>
 <servlet-class>javax.faces.webapp.FacesServlet</servlet-class>
 <load-on-startup>1</load-on-startup>
 </servlet>

 <servlet-mapping>
 <servlet-name>Faces Servlet</servlet-name>
 <url-pattern>*.seam</url-pattern>
 </servlet-mapping>

 <security-constraint>
 <display-name>Restrict raw XHTML Documents</display-name>
 <web-resource-collection>
 <web-resource-name>XHTML</web-resource-name>
 <url-pattern>*.xhtml</url-pattern>
 </web-resource-collection>
 <auth-constraint />
 </security-constraint>

 <session-config>
 <session-timeout>60</session-timeout>
 </session-config>

 <servlet>
 <servlet-name>Document Store Servlet</servlet-name>
 <servlet-class>org.jboss.seam.document.DocumentStoreServlet</servlet-class>
 </servlet>

 <servlet-mapping>
 <servlet-name>Document Store Servlet</servlet-name>
 <url-pattern>*.xls</url-pattern>
 </servlet-mapping>

 <servlet-mapping>
 <servlet-name>Document Store Servlet</servlet-name>
 <url-pattern>*.pdf</url-pattern>
 </servlet-mapping>

 <context-param>
 <param-name>webAppRootKey</param-name>
 <param-value>seam-excel</param-value>
 </context-param>

 <error-page>
 <error-code>400</error-code>
 <location>/home.seam</location>
 </error-page>

 <error-page>
 <error-code>500</error-code>
 <location>/home.seam</location>
 </error-page>

 <error-page>
 <error-code>503</error-code>
 <location>/home.seam</location>
 </error-page>

</web-app>

[gonzalad]

If you have bought Wily Introscope from CA or a similar software, you can activate it on your production server, and see the threads stalled (and their corresponding URI).

If you don't have this software, I don't know.

Also, you could use NEKO filter in your web.xml and the buffer size appears to me to be quite hue (I have 100000 for both facelets.BUFFER_SIZE and com.sun.faces.responseBufferSize), but this won't help with your pb for now.

[gonzalad]

Perhaps https://jira.jboss.org/jira/browse/RF-7248 ?

Otherwise have you checked at RF JIRA ? https://jira.jboss.org/jira/browse/RF

[pramod_bs]

We don't have CA tools. Any open source tools available. Any more ideas? Do you think any kind of profiler will help us to find the URI. Any input is greatly appreciated.

[pramod_bs]

I already looked at https://jira.jboss.org/jira/browse/RF-7248. But couldn't understand what they mean. We are not putting any UI components in session. Do you think if we have any modal in the session will cause this issue.

[gonzalad]

But couldn't understand what they mean. We are not putting any UI components in session
-> I agree with you, from what I understand you shouldn't point to any session scoped managed bean from a binding attribute.

Do you think if we have any modal in the session will cause this issue.
-> don't understand how do you put a rich:modalPanel in session ? with binding attribute ?

[pramod_bs]

"modal" was a typo. We use Seam component model to manage the beans. We do have some session components. I don't think if this could cause this issue. I have some Calendarmodel in the session I think. Do you think it will be a problem?

[gonzalad]

If you have somewhere :
<jsf:acomponent binding="#{mySeamComponent.uiComponent}"/>

Then yes I think this is a pb.
Is it the case ? If yes then you should confirm with someone with better experience than me (sorry !)

[dro_k]

take a look at

http://chiralsoftware.com/launching-a-jboss-seam-site/jboss-seam-problems.seam

[nbelaevski]

Hi,

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6423457 points to the fact that this is behavior observed when HashMap is accessed from multiple threads. So it should be detected who publishes private variable (it is stored in map stored in private field) of UIDataAdaptor class to multiple threads. Usually this can be caused by improper usage of component "binding" attribute sharing the same component instance between threads, or maybe you are creating some components from application code directly? Please check whether something like this is used.

[pramod_bs]

We are not using any binding variables. Nor creating any UI components programatically.
We are using CalendarDataModel in the session. Do you thing it is a potential cause.
What other possibilities are there. Is there anyway I can find using the thread dump which URI is triggering this issue.Please advise.

[nbelaevski]

I think I know where the problem is. Reopening the issue...