Ajax racing & CPU 100% in 3.3.0.GA
Hello,
Periodically we see CPU go to 100% while the memory is very sane at the normal levels (runnig on glassfish). I would greatly appreciate it if somebody can help me alleviate this problem. Taking the java dump shows ajax4jsf restoreChildState and saveChildState are the core of the issue. I maintain state on the server side and global ajax queue is enabled in the web.xml
prstat -mL shows the following when system is hosed
1138 xxxx 85 0.0 0.0 0.0 0.0 0.6 0.0 15 2 552 9 1 java/155 1138 xxxx 76 5.9 0.0 0.0 0.0 0.6 1.0 17 7 533 5K 1 java/14984 1138 xxxx 79 0.0 0.0 0.0 0.0 0.6 0.0 20 2 449 9 1 java/154 1138 xxxx 77 0.0 0.0 0.0 0.0 0.6 0.0 22 3 510 9 1 java/80 1138 xxxx 70 0.0 0.0 0.0 0.0 0.6 0.0 29 4 433 7 1 java/188
The corresponding complete dump is given below
<?xml version='1.0' encoding='UTF-8'?> <hotspot_log version='142 1' process='1138' time_ms='1259633265288'> <vm_version> <name> Java HotSpot(TM) Server VM </name> <release> 1.5.0_14-b03 </release> <info> Java HotSpot(TM) Server VM (1.5.0_14-b03) for solaris-sparc, built on Oct 5 2007 01:41:37 by unknown with unknown Workshop:0x550 </info> </vm_version> <tty> <writer thread='6'/> Full thread dump Java HotSpot(TM) Server VM (1.5.0_14-b03 mixed mode): "RMI TCP Connection(7975)-172.16.1.70" daemon prio=10 tid=0x03202208 nid=0x3a78 runnable [0x5c77f000..0x5c77fa70] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read(BufferedInputStream.java:235) - locked <0xce3dbe40> (a java.io.BufferedInputStream) at java.io.FilterInputStream.read(FilterInputStream.java:66) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) at java.lang.Thread.run(Thread.java:595) "RMI ConnectionExpiration-[172.16.1.70:45016]" daemon prio=10 tid=0x011ea8f0 nid=0x3a77 waiting on condition [0x61e7f000..0x61e7f8f0] at java.lang.Thread.sleep(Native Method) at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446) at java.lang.Thread.run(Thread.java:595) "httpSSLWorkerThread-9001-9" daemon prio=10 tid=0x0085dee0 nid=0x1a73 in Object.wait() [0x59e7f000..0x59e7faf0] at java.lang.Object.wait(Native Method) - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "httpSSLWorkerThread-9001-8" daemon prio=10 tid=0x0087f7e0 nid=0x1a09 in Object.wait() [0x5c27f000..0x5c27f9f0] at java.lang.Object.wait(Native Method) - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "httpSSLWorkerThread-9001-7" daemon prio=10 tid=0x00fdd1b0 nid=0x124e in Object.wait() [0x59b7f000..0x59b7f970] at java.lang.Object.wait(Native Method) - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "Timer-6" daemon prio=10 tid=0x0248ce38 nid=0x29d in Object.wait() [0x61c7f000..0x61c7fbf0] at java.lang.Object.wait(Native Method) - waiting on <0x7b17cbd8> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:474) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x7b17cbd8> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "httpSSLWorkerThread-9001-6" daemon prio=10 tid=0x021c5f08 nid=0xbc runnable [0x61a7c000..0x61a7fc70] at java.util.HashMap.get(HashMap.java:346) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:965) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:943) at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:361) at org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:85) at org.ajax4jsf.model.SequenceDataModel.walk(SequenceDataModel.java:101) at org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1151) at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:106) at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:91) at org.richfaces.renderkit.AbstractRowsRenderer.encodeChildren(AbstractRowsRenderer.java:138) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282) at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215) at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271) at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271) at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892) at javax.faces.render.Renderer.encodeChildren(Renderer.java:137) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892) at com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:245) at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:176) at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100) at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106) at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) "Java2D Disposer" daemon prio=10 tid=0x014f6698 nid=0x9d in Object.wait() [0x6257f000..0x6257fbf0] at java.lang.Object.wait(Native Method) - waiting on <0x7a231270> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked <0x7a231270> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) at sun.java2d.Disposer.run(Disposer.java:125) at java.lang.Thread.run(Thread.java:595) "httpSSLWorkerThread-9001-5" daemon prio=10 tid=0x02181098 nid=0x9c in Object.wait() [0x61b7f000..0x61b7fc70] at java.lang.Object.wait(Native Method) - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "httpSSLWorkerThread-9001-4" daemon prio=10 tid=0x0176b458 nid=0x9b runnable [0x6147c000..0x6147faf0] at java.util.HashMap.get(HashMap.java:346) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:909) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:892) at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:345) at org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:85) at org.ajax4jsf.model.SequenceDataModel.walk(SequenceDataModel.java:101) at org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1151) at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:106) at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:91) at org.richfaces.renderkit.AbstractRowsRenderer.encodeChildren(AbstractRowsRenderer.java:138) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282) at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215) at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271) at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271) at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892) at javax.faces.render.Renderer.encodeChildren(Renderer.java:137) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892) at com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:245) at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:176) at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100) at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106) at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) "httpSSLWorkerThread-9001-3" daemon prio=10 tid=0x00ffb928 nid=0x9a runnable [0x62a7c000..0x62a7fb70] at java.util.HashMap.get(HashMap.java:346) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:965) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982) at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:943) at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:361) at org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:85) at org.ajax4jsf.model.SequenceDataModel.walk(SequenceDataModel.java:101) at org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1151) at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:106) at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:91) at org.richfaces.renderkit.AbstractRowsRenderer.encodeChildren(AbstractRowsRenderer.java:138) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282) at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220) at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215) at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121) at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812) at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:124) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67) at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115) at org.ajax4jsf.renderkit.AjaxContainerRenderer.encodeAjax(AjaxContainerRenderer.java:123) at org.ajax4jsf.component.AjaxViewRoot.encodeAjax(AjaxViewRoot.java:677) at org.ajax4jsf.component.AjaxViewRoot.encodeChildren(AjaxViewRoot.java:548) at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886) at com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:245) at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:176) at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100) at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176) at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106) at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) "httpSSLWorkerThread-9001-2" daemon prio=10 tid=0x00306b28 nid=0x99 in Object.wait() [0x62b7f000..0x62b7f9f0] at java.lang.Object.wait(Native Method) - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "JMX server connection timeout 114" daemon prio=10 tid=0x020c4828 nid=0x73 in Object.wait() [0x6287f000..0x6287faf0] at java.lang.Object.wait(Native Method) - waiting on <0x78d6ebe0> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) - locked <0x78d6ebe0> (a [I) at java.lang.Thread.run(Thread.java:595) "JMX server connection timeout 112" daemon prio=10 tid=0x008a24f0 nid=0x71 in Object.wait() [0x6297f000..0x6297f9f0] at java.lang.Object.wait(Native Method) - waiting on <0x78d6e498> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) - locked <0x78d6e498> (a [I) at java.lang.Thread.run(Thread.java:595) "JMX server connection timeout 98" daemon prio=10 tid=0x00e1def8 nid=0x63 in Object.wait() [0x6227f000..0x6227faf0] at java.lang.Object.wait(Native Method) - waiting on <0x78d6f220> (a [I) at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150) - locked <0x78d6f220> (a [I) at java.lang.Thread.run(Thread.java:595) "DestroyJavaVM" prio=10 tid=0x000b2b18 nid=0x1 waiting on condition [0x00000000..0xffbfd9b8] "Thread-25" daemon prio=1 tid=0x018c2710 nid=0x60 waiting on condition [0x61d7f000..0x61d7fa70] at java.lang.Thread.sleep(Native Method) at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154) at java.lang.Thread.run(Thread.java:595) "Timer-5" daemon prio=10 tid=0x01d45ab8 nid=0x5c in Object.wait() [0x6207f000..0x6207fc70] at java.lang.Object.wait(Native Method) - waiting on <0x7898f148> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x7898f148> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=10 tid=0x0141d8b0 nid=0x59 waiting on condition [0x6237f000..0x6237f9f0] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800) at java.lang.Thread.run(Thread.java:595) "Timer-4" daemon prio=10 tid=0x01a03308 nid=0x58 in Object.wait() [0x608ff000..0x608ffa70] at java.lang.Object.wait(Native Method) - waiting on <0x7a1e7868> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x7a1e7868> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/GBWeb]]" daemon prio=10 tid=0x01768e30 nid=0x57 waiting on condition [0x5d07f000..0x5d07f8f0] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800) at java.lang.Thread.run(Thread.java:595) "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]" daemon prio=10 tid=0x016f0ea8 nid=0x56 waiting on condition [0x5d17f000..0x5d17f970] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800) at java.lang.Thread.run(Thread.java:595) "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]" daemon prio=10 tid=0x01707750 nid=0x55 waiting on condition [0x5d27f000..0x5d27fbf0] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800) at java.lang.Thread.run(Thread.java:595) "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]" daemon prio=10 tid=0x01875040 nid=0x54 waiting on condition [0x5d37f000..0x5d37fc70] at java.lang.Thread.sleep(Native Method) at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800) at java.lang.Thread.run(Thread.java:595) "httpSSLWorkerThread-7001-1" daemon prio=10 tid=0x0047d310 nid=0x53 in Object.wait() [0x5e5ff000..0x5e5ffaf0] at java.lang.Object.wait(Native Method) - waiting on <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "httpSSLWorkerThread-9001-1" daemon prio=10 tid=0x00db9158 nid=0x52 in Object.wait() [0x5e6ff000..0x5e6ffb70] at java.lang.Object.wait(Native Method) - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "httpSSLWorkerThread-7001-0" daemon prio=10 tid=0x00467480 nid=0x51 in Object.wait() [0x5e7ff000..0x5e7ff9f0] at java.lang.Object.wait(Native Method) - waiting on <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline) at java.lang.Object.wait(Object.java:474) at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291) - locked <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104) "httpSSLWorkerThread-9001-0" daemon prio=10 tid=0x0163d8a0 nid=0x50 runnable [0x5e8fd000..0x5e8ffa70] at java.util.HashMap.get(HashMap.java:346) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:909) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924) at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:892) at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:345) at org.ajax4jsf.component.UIDataAdaptor.iterate(UIDataAdaptor.java:1034) at org.ajax4jsf.component.UIDataAdaptor.processDecodes(UIDataAdaptor.java:1158) at org.ajax4jsf.component.UIDataAdaptor.processDecodes(UIDataAdaptor.java:1168) at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001) at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001) at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001) at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001) at javax.faces.component.UIForm.processDecodes(UIForm.java:203) at org.ajax4jsf.component.AjaxViewRoot$1.invokeContextCallback(AjaxViewRoot.java:392) at org.ajax4jsf.component.AjaxViewRoot.processPhase(AjaxViewRoot.java:238) at org.ajax4jsf.component.AjaxViewRoot.processDecodes(AjaxViewRoot.java:409) at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:101) at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251) at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117) at javax.faces.webapp.FacesServlet.service(FacesServlet.java:244) at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178) at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290) at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390) at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106) "SelectorThread-7001" prio=10 tid=0x01105ad0 nid=0x4f runnable [0x5e9ff000..0x5e9ff8f0] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x7a1d6350> (a sun.nio.ch.Util$1) - locked <0x7a1d6340> (a java.util.Collections$UnmodifiableSet) - locked <0x7941a3e8> (a sun.nio.ch.DevPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1337) at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284) - locked <0x7941a278> (a [Ljava.lang.Object;) at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247) at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223) "SelectorThread-9001" prio=10 tid=0x01875fe8 nid=0x4e runnable [0x5eaff000..0x5eaff970] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x78dd41f0> (a sun.nio.ch.Util$1) - locked <0x78dd41e0> (a java.util.Collections$UnmodifiableSet) - locked <0x78dd2370> (a sun.nio.ch.DevPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1337) at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284) - locked <0x78dd2200> (a [Ljava.lang.Object;) at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247) at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223) "EEHADBHealthChecker" daemon prio=10 tid=0x01ec6760 nid=0x4d waiting on condition [0x600ff000..0x600ffbf0] at java.lang.Thread.sleep(Native Method) at com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.threadSleep(EEHADBHealthChecker.java:467) at com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.run(EEHADBHealthChecker.java:479) at java.lang.Thread.run(Thread.java:595) "Thread-20" daemon prio=10 tid=0x0112f930 nid=0x4c waiting on condition [0x5ebff000..0x5ebffc70] at java.lang.Thread.sleep(Native Method) at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154) at java.lang.Thread.run(Thread.java:595) "RMI RenewClean-[172.16.1.70:45016]" daemon prio=10 tid=0x017865f8 nid=0x4a in Object.wait() [0x5edff000..0x5edffb70] at java.lang.Object.wait(Native Method) - waiting on <0x79419420> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked <0x79419420> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501) at java.lang.Thread.run(Thread.java:595) "iMQReadChannel-4" daemon prio=10 tid=0x008f7de0 nid=0x48 runnable [0x5efff000..0x5efffa70] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:313) - locked <0x78db1e90> (a java.io.BufferedInputStream) at com.sun.messaging.jmq.io.ReadOnlyPacket.readFully(ReadOnlyPacket.java:271) at com.sun.messaging.jmq.io.ReadOnlyPacket.readFixedHeader(ReadOnlyPacket.java:191) at com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:151) at com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:82) - locked <0x7a312418> (a com.sun.messaging.jmq.io.ReadWritePacket) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1738) at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1160) at java.lang.Thread.run(Thread.java:595) "imqConnectionFlowControl-4" daemon prio=10 tid=0x00a551d8 nid=0x47 in Object.wait() [0x5f0ff000..0x5f0ff8f0] at java.lang.Object.wait(Native Method) - waiting on <0x78da3190> (a com.sun.messaging.jmq.jmsclient.FlowControl) at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301) - locked <0x78da3190> (a com.sun.messaging.jmq.jmsclient.FlowControl) at java.lang.Thread.run(Thread.java:595) "imqConsumerReader-3-2254374079623271680-0" daemon prio=10 tid=0x00dffd70 nid=0x46 in Object.wait() [0x5f1ff000..0x5f1ff970] at java.lang.Object.wait(Native Method) - waiting on <0x78d59808> (a com.sun.messaging.jmq.jmsclient.SessionQueue) at com.sun.messaging.jmq.jmsclient.SessionQueue.dequeueWait(SessionQueue.java:236) - locked <0x78d59808> (a com.sun.messaging.jmq.jmsclient.SessionQueue) at com.sun.messaging.jmq.jmsclient.ConsumerReader.run(ConsumerReader.java:161) at java.lang.Thread.run(Thread.java:595) "iMQReadChannel-3" daemon prio=10 tid=0x010ae3d8 nid=0x45 runnable [0x5f2ff000..0x5f2ffbf0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:256) at java.io.BufferedInputStream.read(BufferedInputStream.java:313) - locked <0x78d5a030> (a java.io.BufferedInputStream) at com.sun.messaging.jmq.io.ReadOnlyPacket.readFully(ReadOnlyPacket.java:271) at com.sun.messaging.jmq.io.ReadOnlyPacket.readFixedHeader(ReadOnlyPacket.java:191) at com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:151) at com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:82) - locked <0x7a312a48> (a com.sun.messaging.jmq.io.ReadWritePacket) at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1738) at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1160) at java.lang.Thread.run(Thread.java:595) "imqConnectionFlowControl-3" daemon prio=10 tid=0x008ea588 nid=0x44 in Object.wait() [0x5f3ff000..0x5f3ffc70] at java.lang.Object.wait(Native Method) - waiting on <0x78d66c58> (a com.sun.messaging.jmq.jmsclient.FlowControl) at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301) - locked <0x78d66c58> (a com.sun.messaging.jmq.jmsclient.FlowControl) at java.lang.Thread.run(Thread.java:595) "process reaper" daemon prio=10 tid=0x00e27198 nid=0x43 runnable [0x5f4ff000..0x5f4ffaf0] at java.lang.UNIXProcess.waitForProcessExit(Native Method) at java.lang.UNIXProcess.access$900(UNIXProcess.java:17) at java.lang.UNIXProcess$2$1.run(UNIXProcess.java:86) "RMI RenewClean-[172.16.1.90:33624]" daemon prio=10 tid=0x00575240 nid=0x3d in Object.wait() [0x5fcff000..0x5fcffbf0] at java.lang.Object.wait(Native Method) - waiting on <0x794198a0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked <0x794198a0> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501) at java.lang.Thread.run(Thread.java:595) "RMI TCP Accept-0" daemon prio=10 tid=0x00fc5ba8 nid=0x3b runnable [0x5feff000..0x5feffaf0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x78d5bf98> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340) at java.lang.Thread.run(Thread.java:595) "RMI TCP Accept-38688" daemon prio=10 tid=0x00d5e2d0 nid=0x3a runnable [0x5ffff000..0x5ffffb70] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x78d5b090> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340) at java.lang.Thread.run(Thread.java:595) "Thread-10" prio=10 tid=0x010ef658 nid=0x37 in Object.wait() [0x601ff000..0x601ff8f0] at java.lang.Object.wait(Native Method) - waiting on <0x78953ff0> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive) at java.lang.Object.wait(Object.java:474) at com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:857) - locked <0x78953ff0> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive) "p: thread-pool-1; w: 2" daemon prio=10 tid=0x01107470 nid=0x36 runnable [0x602ff000..0x602ffbf0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x78d381f8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259) at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:250) at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107) at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555) "p: thread-pool-1; w: 1" daemon prio=10 tid=0x00ca0610 nid=0x35 runnable [0x603ff000..0x603ff970] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x78d382a8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259) at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:250) at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107) at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555) "SelectorThread" daemon prio=10 tid=0x00b8a160 nid=0x34 runnable [0x604ff000..0x604ffc70] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x78d386f0> (a sun.nio.ch.Util$1) - locked <0x78d386e0> (a java.util.Collections$UnmodifiableSet) - locked <0x78d384a8> (a sun.nio.ch.DevPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283) "Timer-3" daemon prio=10 tid=0x00bb7620 nid=0x33 in Object.wait() [0x605ff000..0x605ffaf0] at java.lang.Object.wait(Native Method) - waiting on <0x793ffbc8> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:474) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x793ffbc8> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "Timer-2" prio=10 tid=0x00b627a0 nid=0x32 in Object.wait() [0x606ff000..0x606ffb70] at java.lang.Object.wait(Native Method) - waiting on <0x793ffc50> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <0x793ffc50> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "Thread-6" prio=10 tid=0x00cf5808 nid=0x31 waiting on condition [0x607ff000..0x607ff9f0] at java.lang.Thread.sleep(Native Method) at com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:151) at java.lang.Thread.run(Thread.java:595) "RMI RenewClean-[172.16.1.90:33598,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory@1d2bb9f]" daemon prio=10 tid=0x00b6d030 nid=0x2e in Object.wait() [0x60aff000..0x60aff970] at java.lang.Object.wait(Native Method) - waiting on <0x79400eb0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked <0x79400eb0> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501) at java.lang.Thread.run(Thread.java:595) "Timer-1" prio=10 tid=0x006f5f18 nid=0x2d in Object.wait() [0x67b7f000..0x67b7fbf0] at java.lang.Object.wait(Native Method) - waiting on <0x79400368> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:474) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x79400368> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "YJPAgent-RequestListener" daemon prio=1 tid=0x00d56ff8 nid=0x2c runnable [0x6107f000..0x6107fc70] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x79400840> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at com.yourkit.runtime.Core$4.run(Core.java:518) at java.lang.Thread.run(Thread.java:595) "YJPAgent-Telemetry" daemon prio=10 tid=0x0077c5a8 nid=0x2b waiting on condition [0x6127f000..0x6127faf0] at java.lang.Thread.sleep(Native Method) at com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284) "RMI LeaseChecker" daemon prio=10 tid=0x00d57568 nid=0x2a waiting on condition [0x6137f000..0x6137fb70] at java.lang.Thread.sleep(Native Method) at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310) at java.lang.Thread.run(Thread.java:595) "GC Daemon" daemon prio=2 tid=0x00a3d418 nid=0x28 in Object.wait() [0x6157f000..0x6157fa70] at java.lang.Object.wait(Native Method) - waiting on <0x784072b8> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:100) - locked <0x784072b8> (a sun.misc.GC$LatencyLock) "RMI Reaper" prio=10 tid=0x00a3d9b0 nid=0x27 in Object.wait() [0x6167f000..0x6167f8f0] at java.lang.Object.wait(Native Method) - waiting on <0x7840b1d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked <0x7840b1d8> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336) at java.lang.Thread.run(Thread.java:595) "Timer-0" daemon prio=10 tid=0x00744d20 nid=0x26 in Object.wait() [0x6177f000..0x6177f970] at java.lang.Object.wait(Native Method) - waiting on <0x78400230> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:474) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <0x78400230> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "RMI TCP Accept-0" daemon prio=10 tid=0x00e13228 nid=0x25 runnable [0x6187f000..0x6187fbf0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) - locked <0x78477628> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340) at java.lang.Thread.run(Thread.java:595) "Low Memory Detector" daemon prio=10 tid=0x0017d4c0 nid=0x10 runnable [0x00000000..0x00000000] "CompilerThread1" daemon prio=10 tid=0x00217640 nid=0xf waiting on condition [0x00000000..0x6757e9a4] "CompilerThread0" daemon prio=10 tid=0x001f3288 nid=0xe waiting on condition [0x00000000..0x6767ea2c] "AdapterThread" daemon prio=10 tid=0x00225b38 nid=0xd waiting on condition [0x00000000..0x00000000] "Signal Dispatcher" daemon prio=10 tid=0x00212ed8 nid=0xc waiting on condition [0x00000000..0x00000000] "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00220290 nid=0xb runnable [0x00000000..0x00000000] "YJPAgent-CPUSampler" daemon prio=10 tid=0x00223290 nid=0xa runnable [0x00000000..0x00000000] "Finalizer" daemon prio=10 tid=0x001974a8 nid=0x8 in Object.wait() [0x6817f000..0x6817fa70] at java.lang.Object.wait(Native Method) - waiting on <0x78407560> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) - locked <0x78407560> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x00176870 nid=0x7 in Object.wait() [0x6827f000..0x6827f8f0] at java.lang.Object.wait(Native Method) - waiting on <0x78403598> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x78403598> (a java.lang.ref.Reference$Lock) "VM Thread" prio=10 tid=0x001739c0 nid=0x6 runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x00137388 nid=0x2 runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x00137dc8 nid=0x3 runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x00139010 nid=0x4 runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x00139a50 nid=0x5 runnable "VM Periodic Task Thread" prio=10 tid=0x0012ea80 nid=0x11 runnable
Thanks,
Ravi Kiran Bhaskar
