6 Replies Latest reply on Dec 11, 2009 11:37 AM by Jay Balunas

    Ajax racing & CPU 100% in 3.3.0.GA

    Ravi Kiran Bhaskar Newbie

      Hello,
      Periodically we see CPU go to 100% while the memory is very sane at the normal levels (runnig on glassfish). I would greatly appreciate it if somebody can help me alleviate this problem. Taking the java dump shows ajax4jsf restoreChildState and saveChildState are the core of the issue. I maintain state on the server side and global ajax queue is enabled in the web.xml

      prstat -mL shows the following when system is hosed

       1138 xxxx 85 0.0 0.0 0.0 0.0 0.6 0.0 15 2 552 9 1 java/155
       1138 xxxx 76 5.9 0.0 0.0 0.0 0.6 1.0 17 7 533 5K 1 java/14984
       1138 xxxx 79 0.0 0.0 0.0 0.0 0.6 0.0 20 2 449 9 1 java/154
       1138 xxxx 77 0.0 0.0 0.0 0.0 0.6 0.0 22 3 510 9 1 java/80
       1138 xxxx 70 0.0 0.0 0.0 0.0 0.6 0.0 29 4 433 7 1 java/188
      


      The corresponding complete dump is given below

      <?xml version='1.0' encoding='UTF-8'?>
      <hotspot_log version='142 1' process='1138' time_ms='1259633265288'>
      <vm_version>
      <name>
      Java HotSpot(TM) Server VM
      </name>
      <release>
      1.5.0_14-b03
      </release>
      <info>
      Java HotSpot(TM) Server VM (1.5.0_14-b03) for solaris-sparc, built on Oct 5 2007 01:41:37 by unknown with unknown Workshop:0x550
      </info>
      </vm_version>
      <tty>
      <writer thread='6'/>
      Full thread dump Java HotSpot(TM) Server VM (1.5.0_14-b03 mixed mode):
      
      "RMI TCP Connection(7975)-172.16.1.70" daemon prio=10 tid=0x03202208 nid=0x3a78 runnable [0x5c77f000..0x5c77fa70]
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
       - locked <0xce3dbe40> (a java.io.BufferedInputStream)
       at java.io.FilterInputStream.read(FilterInputStream.java:66)
       at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:448)
       at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
       at java.lang.Thread.run(Thread.java:595)
      
      "RMI ConnectionExpiration-[172.16.1.70:45016]" daemon prio=10 tid=0x011ea8f0 nid=0x3a77 waiting on condition [0x61e7f000..0x61e7f8f0]
       at java.lang.Thread.sleep(Native Method)
       at sun.rmi.transport.tcp.TCPChannel$Reaper.run(TCPChannel.java:446)
       at java.lang.Thread.run(Thread.java:595)
      
      "httpSSLWorkerThread-9001-9" daemon prio=10 tid=0x0085dee0 nid=0x1a73 in Object.wait() [0x59e7f000..0x59e7faf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "httpSSLWorkerThread-9001-8" daemon prio=10 tid=0x0087f7e0 nid=0x1a09 in Object.wait() [0x5c27f000..0x5c27f9f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "httpSSLWorkerThread-9001-7" daemon prio=10 tid=0x00fdd1b0 nid=0x124e in Object.wait() [0x59b7f000..0x59b7f970]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "Timer-6" daemon prio=10 tid=0x0248ce38 nid=0x29d in Object.wait() [0x61c7f000..0x61c7fbf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7b17cbd8> (a java.util.TaskQueue)
       at java.lang.Object.wait(Object.java:474)
       at java.util.TimerThread.mainLoop(Timer.java:483)
       - locked <0x7b17cbd8> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "httpSSLWorkerThread-9001-6" daemon prio=10 tid=0x021c5f08 nid=0xbc runnable [0x61a7c000..0x61a7fc70]
       at java.util.HashMap.get(HashMap.java:346)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:965)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:943)
       at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:361)
       at org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:85)
       at org.ajax4jsf.model.SequenceDataModel.walk(SequenceDataModel.java:101)
       at org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1151)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:106)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:91)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeChildren(AbstractRowsRenderer.java:138)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282)
       at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262)
       at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220)
       at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215)
       at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271)
       at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271)
       at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892)
       at javax.faces.render.Renderer.encodeChildren(Renderer.java:137)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892)
       at com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:245)
       at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:176)
       at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100)
       at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176)
       at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
       at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
       at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
       at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
       at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178)
       at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
       at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390)
       at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
       at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
       at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
       at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
      
      "Java2D Disposer" daemon prio=10 tid=0x014f6698 nid=0x9d in Object.wait() [0x6257f000..0x6257fbf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7a231270> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
       - locked <0x7a231270> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
       at sun.java2d.Disposer.run(Disposer.java:125)
       at java.lang.Thread.run(Thread.java:595)
      
      "httpSSLWorkerThread-9001-5" daemon prio=10 tid=0x02181098 nid=0x9c in Object.wait() [0x61b7f000..0x61b7fc70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "httpSSLWorkerThread-9001-4" daemon prio=10 tid=0x0176b458 nid=0x9b runnable [0x6147c000..0x6147faf0]
       at java.util.HashMap.get(HashMap.java:346)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:909)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:892)
       at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:345)
       at org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:85)
       at org.ajax4jsf.model.SequenceDataModel.walk(SequenceDataModel.java:101)
       at org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1151)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:106)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:91)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeChildren(AbstractRowsRenderer.java:138)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282)
       at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262)
       at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220)
       at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215)
       at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271)
       at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at com.sun.faces.renderkit.html_basic.HtmlBasicRenderer.encodeRecursive(HtmlBasicRenderer.java:271)
       at com.sun.faces.renderkit.html_basic.GridRenderer.encodeChildren(GridRenderer.java:242)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892)
       at javax.faces.render.Renderer.encodeChildren(Renderer.java:137)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:892)
       at com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:245)
       at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:176)
       at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100)
       at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176)
       at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
       at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
       at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
       at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
       at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178)
       at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
       at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390)
       at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
       at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
       at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
       at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
      
      "httpSSLWorkerThread-9001-3" daemon prio=10 tid=0x00ffb928 nid=0x9a runnable [0x62a7c000..0x62a7fb70]
       at java.util.HashMap.get(HashMap.java:346)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:965)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:982)
       at org.ajax4jsf.component.UIDataAdaptor.restoreChildState(UIDataAdaptor.java:943)
       at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:361)
       at org.richfaces.renderkit.AbstractRowsRenderer.process(AbstractRowsRenderer.java:85)
       at org.ajax4jsf.model.SequenceDataModel.walk(SequenceDataModel.java:101)
       at org.ajax4jsf.component.UIDataAdaptor.walk(UIDataAdaptor.java:1151)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:106)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeRows(AbstractRowsRenderer.java:91)
       at org.richfaces.renderkit.AbstractRowsRenderer.encodeChildren(AbstractRowsRenderer.java:138)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282)
       at org.ajax4jsf.renderkit.RendererBase.renderChildren(RendererBase.java:262)
       at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:220)
       at org.richfaces.renderkit.html.PanelRenderer.doEncodeChildren(PanelRenderer.java:215)
       at org.ajax4jsf.renderkit.RendererBase.encodeChildren(RendererBase.java:121)
       at javax.faces.component.UIComponentBase.encodeChildren(UIComponentBase.java:812)
       at org.ajax4jsf.renderkit.RendererBase.renderChild(RendererBase.java:282)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:124)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxChildren(AjaxChildrenRenderer.java:67)
       at org.ajax4jsf.renderkit.AjaxChildrenRenderer.encodeAjaxComponent(AjaxChildrenRenderer.java:115)
       at org.ajax4jsf.renderkit.AjaxContainerRenderer.encodeAjax(AjaxContainerRenderer.java:123)
       at org.ajax4jsf.component.AjaxViewRoot.encodeAjax(AjaxViewRoot.java:677)
       at org.ajax4jsf.component.AjaxViewRoot.encodeChildren(AjaxViewRoot.java:548)
       at javax.faces.component.UIComponent.encodeAll(UIComponent.java:886)
       at com.sun.faces.application.ViewHandlerImpl.doRenderView(ViewHandlerImpl.java:245)
       at com.sun.faces.application.ViewHandlerImpl.renderView(ViewHandlerImpl.java:176)
       at org.ajax4jsf.application.ViewHandlerWrapper.renderView(ViewHandlerWrapper.java:100)
       at org.ajax4jsf.application.AjaxViewHandler.renderView(AjaxViewHandler.java:176)
       at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:106)
       at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
       at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:144)
       at javax.faces.webapp.FacesServlet.service(FacesServlet.java:245)
       at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178)
       at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
       at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390)
       at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
       at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
       at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
       at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
      
      "httpSSLWorkerThread-9001-2" daemon prio=10 tid=0x00306b28 nid=0x99 in Object.wait() [0x62b7f000..0x62b7f9f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "JMX server connection timeout 114" daemon prio=10 tid=0x020c4828 nid=0x73 in Object.wait() [0x6287f000..0x6287faf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d6ebe0> (a [I)
       at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
       - locked <0x78d6ebe0> (a [I)
       at java.lang.Thread.run(Thread.java:595)
      
      "JMX server connection timeout 112" daemon prio=10 tid=0x008a24f0 nid=0x71 in Object.wait() [0x6297f000..0x6297f9f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d6e498> (a [I)
       at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
       - locked <0x78d6e498> (a [I)
       at java.lang.Thread.run(Thread.java:595)
      
      "JMX server connection timeout 98" daemon prio=10 tid=0x00e1def8 nid=0x63 in Object.wait() [0x6227f000..0x6227faf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d6f220> (a [I)
       at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
       - locked <0x78d6f220> (a [I)
       at java.lang.Thread.run(Thread.java:595)
      
      "DestroyJavaVM" prio=10 tid=0x000b2b18 nid=0x1 waiting on condition [0x00000000..0xffbfd9b8]
      
      "Thread-25" daemon prio=1 tid=0x018c2710 nid=0x60 waiting on condition [0x61d7f000..0x61d7fa70]
       at java.lang.Thread.sleep(Native Method)
       at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
       at java.lang.Thread.run(Thread.java:595)
      
      "Timer-5" daemon prio=10 tid=0x01d45ab8 nid=0x5c in Object.wait() [0x6207f000..0x6207fc70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7898f148> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x7898f148> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv]]" daemon prio=10 tid=0x0141d8b0 nid=0x59 waiting on condition [0x6237f000..0x6237f9f0]
       at java.lang.Thread.sleep(Native Method)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
       at java.lang.Thread.run(Thread.java:595)
      
      "Timer-4" daemon prio=10 tid=0x01a03308 nid=0x58 in Object.wait() [0x608ff000..0x608ffa70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7a1e7868> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x7a1e7868> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/GBWeb]]" daemon prio=10 tid=0x01768e30 nid=0x57 waiting on condition [0x5d07f000..0x5d07f8f0]
       at java.lang.Thread.sleep(Native Method)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
       at java.lang.Thread.run(Thread.java:595)
      
      "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__wstx-services]]" daemon prio=10 tid=0x016f0ea8 nid=0x56 waiting on condition [0x5d17f000..0x5d17f970]
       at java.lang.Thread.sleep(Native Method)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
       at java.lang.Thread.run(Thread.java:595)
      
      "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[/__JWSappclients]]" daemon prio=10 tid=0x01707750 nid=0x55 waiting on condition [0x5d27f000..0x5d27fbf0]
       at java.lang.Thread.sleep(Native Method)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
       at java.lang.Thread.run(Thread.java:595)
      
      "ContainerBackgroundProcessor[StandardEngine[com.sun.appserv].StandardHost[server].StandardContext[]]" daemon prio=10 tid=0x01875040 nid=0x54 waiting on condition [0x5d37f000..0x5d37fc70]
       at java.lang.Thread.sleep(Native Method)
       at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1800)
       at java.lang.Thread.run(Thread.java:595)
      
      "httpSSLWorkerThread-7001-1" daemon prio=10 tid=0x0047d310 nid=0x53 in Object.wait() [0x5e5ff000..0x5e5ffaf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "httpSSLWorkerThread-9001-1" daemon prio=10 tid=0x00db9158 nid=0x52 in Object.wait() [0x5e6ff000..0x5e6ffb70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x78d9bf30> (a com.sun.enterprise.web.portunif.PortUnificationPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "httpSSLWorkerThread-7001-0" daemon prio=10 tid=0x00467480 nid=0x51 in Object.wait() [0x5e7ff000..0x5e7ff9f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.enterprise.web.connector.grizzly.LinkedListPipeline.getTask(LinkedListPipeline.java:291)
       - locked <0x7941a450> (a com.sun.enterprise.web.connector.grizzly.ssl.SSLPipeline)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:104)
      
      "httpSSLWorkerThread-9001-0" daemon prio=10 tid=0x0163d8a0 nid=0x50 runnable [0x5e8fd000..0x5e8ffa70]
       at java.util.HashMap.get(HashMap.java:346)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:909)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:924)
       at org.ajax4jsf.component.UIDataAdaptor.saveChildState(UIDataAdaptor.java:892)
       at org.ajax4jsf.component.UIDataAdaptor.setRowKey(UIDataAdaptor.java:345)
       at org.ajax4jsf.component.UIDataAdaptor.iterate(UIDataAdaptor.java:1034)
       at org.ajax4jsf.component.UIDataAdaptor.processDecodes(UIDataAdaptor.java:1158)
       at org.ajax4jsf.component.UIDataAdaptor.processDecodes(UIDataAdaptor.java:1168)
       at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001)
       at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001)
       at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001)
       at javax.faces.component.UIComponentBase.processDecodes(UIComponentBase.java:1001)
       at javax.faces.component.UIForm.processDecodes(UIForm.java:203)
       at org.ajax4jsf.component.AjaxViewRoot$1.invokeContextCallback(AjaxViewRoot.java:392)
       at org.ajax4jsf.component.AjaxViewRoot.processPhase(AjaxViewRoot.java:238)
       at org.ajax4jsf.component.AjaxViewRoot.processDecodes(AjaxViewRoot.java:409)
       at com.sun.faces.lifecycle.ApplyRequestValuesPhase.execute(ApplyRequestValuesPhase.java:101)
       at com.sun.faces.lifecycle.LifecycleImpl.phase(LifecycleImpl.java:251)
       at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:117)
       at javax.faces.webapp.FacesServlet.service(FacesServlet.java:244)
       at org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.ajax4jsf.webapp.BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:178)
       at org.ajax4jsf.webapp.BaseFilter.handleRequest(BaseFilter.java:290)
       at org.ajax4jsf.webapp.BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:390)
       at org.ajax4jsf.webapp.BaseFilter.doFilter(BaseFilter.java:517)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288)
       at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632)
       at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577)
       at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571)
       at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080)
       at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568)
       at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263)
       at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214)
       at com.sun.enterprise.web.portunif.PortUnificationPipeline$PUTask.doTask(PortUnificationPipeline.java:380)
       at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)
       at com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)
      
      "SelectorThread-7001" prio=10 tid=0x01105ad0 nid=0x4f runnable [0x5e9ff000..0x5e9ff8f0]
       at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
       at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
       at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x7a1d6350> (a sun.nio.ch.Util$1)
       - locked <0x7a1d6340> (a java.util.Collections$UnmodifiableSet)
       - locked <0x7941a3e8> (a sun.nio.ch.DevPollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1337)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284)
       - locked <0x7941a278> (a [Ljava.lang.Object;)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)
      
      "SelectorThread-9001" prio=10 tid=0x01875fe8 nid=0x4e runnable [0x5eaff000..0x5eaff970]
       at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
       at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
       at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x78dd41f0> (a sun.nio.ch.Util$1)
       - locked <0x78dd41e0> (a java.util.Collections$UnmodifiableSet)
       - locked <0x78dd2370> (a sun.nio.ch.DevPollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.doSelect(SelectorThread.java:1337)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.startListener(SelectorThread.java:1284)
       - locked <0x78dd2200> (a [Ljava.lang.Object;)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.startEndpoint(SelectorThread.java:1247)
       at com.sun.enterprise.web.connector.grizzly.SelectorThread.run(SelectorThread.java:1223)
      
      "EEHADBHealthChecker" daemon prio=10 tid=0x01ec6760 nid=0x4d waiting on condition [0x600ff000..0x600ffbf0]
       at java.lang.Thread.sleep(Native Method)
       at com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.threadSleep(EEHADBHealthChecker.java:467)
       at com.sun.enterprise.ee.web.sessmgmt.EEHADBHealthChecker.run(EEHADBHealthChecker.java:479)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-20" daemon prio=10 tid=0x0112f930 nid=0x4c waiting on condition [0x5ebff000..0x5ebffc70]
       at java.lang.Thread.sleep(Native Method)
       at com.sun.jmx.remote.internal.ClientCommunicatorAdmin$Checker.run(ClientCommunicatorAdmin.java:154)
       at java.lang.Thread.run(Thread.java:595)
      
      "RMI RenewClean-[172.16.1.70:45016]" daemon prio=10 tid=0x017865f8 nid=0x4a in Object.wait() [0x5edff000..0x5edffb70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x79419420> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
       - locked <0x79419420> (a java.lang.ref.ReferenceQueue$Lock)
       at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
       at java.lang.Thread.run(Thread.java:595)
      
      "iMQReadChannel-4" daemon prio=10 tid=0x008f7de0 nid=0x48 runnable [0x5efff000..0x5efffa70]
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
       at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
       - locked <0x78db1e90> (a java.io.BufferedInputStream)
       at com.sun.messaging.jmq.io.ReadOnlyPacket.readFully(ReadOnlyPacket.java:271)
       at com.sun.messaging.jmq.io.ReadOnlyPacket.readFixedHeader(ReadOnlyPacket.java:191)
       at com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:151)
       at com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:82)
       - locked <0x7a312418> (a com.sun.messaging.jmq.io.ReadWritePacket)
       at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1738)
       at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1160)
       at java.lang.Thread.run(Thread.java:595)
      
      "imqConnectionFlowControl-4" daemon prio=10 tid=0x00a551d8 nid=0x47 in Object.wait() [0x5f0ff000..0x5f0ff8f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78da3190> (a com.sun.messaging.jmq.jmsclient.FlowControl)
       at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301)
       - locked <0x78da3190> (a com.sun.messaging.jmq.jmsclient.FlowControl)
       at java.lang.Thread.run(Thread.java:595)
      
      "imqConsumerReader-3-2254374079623271680-0" daemon prio=10 tid=0x00dffd70 nid=0x46 in Object.wait() [0x5f1ff000..0x5f1ff970]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d59808> (a com.sun.messaging.jmq.jmsclient.SessionQueue)
       at com.sun.messaging.jmq.jmsclient.SessionQueue.dequeueWait(SessionQueue.java:236)
       - locked <0x78d59808> (a com.sun.messaging.jmq.jmsclient.SessionQueue)
       at com.sun.messaging.jmq.jmsclient.ConsumerReader.run(ConsumerReader.java:161)
       at java.lang.Thread.run(Thread.java:595)
      
      "iMQReadChannel-3" daemon prio=10 tid=0x010ae3d8 nid=0x45 runnable [0x5f2ff000..0x5f2ffbf0]
       at java.net.SocketInputStream.socketRead0(Native Method)
       at java.net.SocketInputStream.read(SocketInputStream.java:129)
       at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
       at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
       at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
       - locked <0x78d5a030> (a java.io.BufferedInputStream)
       at com.sun.messaging.jmq.io.ReadOnlyPacket.readFully(ReadOnlyPacket.java:271)
       at com.sun.messaging.jmq.io.ReadOnlyPacket.readFixedHeader(ReadOnlyPacket.java:191)
       at com.sun.messaging.jmq.io.ReadOnlyPacket.readPacket(ReadOnlyPacket.java:151)
       at com.sun.messaging.jmq.io.ReadWritePacket.readPacket(ReadWritePacket.java:82)
       - locked <0x7a312a48> (a com.sun.messaging.jmq.io.ReadWritePacket)
       at com.sun.messaging.jmq.jmsclient.ProtocolHandler.readPacket(ProtocolHandler.java:1738)
       at com.sun.messaging.jmq.jmsclient.ReadChannel.run(ReadChannel.java:1160)
       at java.lang.Thread.run(Thread.java:595)
      
      "imqConnectionFlowControl-3" daemon prio=10 tid=0x008ea588 nid=0x44 in Object.wait() [0x5f3ff000..0x5f3ffc70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78d66c58> (a com.sun.messaging.jmq.jmsclient.FlowControl)
       at com.sun.messaging.jmq.jmsclient.FlowControl.run(FlowControl.java:301)
       - locked <0x78d66c58> (a com.sun.messaging.jmq.jmsclient.FlowControl)
       at java.lang.Thread.run(Thread.java:595)
      
      "process reaper" daemon prio=10 tid=0x00e27198 nid=0x43 runnable [0x5f4ff000..0x5f4ffaf0]
       at java.lang.UNIXProcess.waitForProcessExit(Native Method)
       at java.lang.UNIXProcess.access$900(UNIXProcess.java:17)
       at java.lang.UNIXProcess$2$1.run(UNIXProcess.java:86)
      
      "RMI RenewClean-[172.16.1.90:33624]" daemon prio=10 tid=0x00575240 nid=0x3d in Object.wait() [0x5fcff000..0x5fcffbf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x794198a0> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
       - locked <0x794198a0> (a java.lang.ref.ReferenceQueue$Lock)
       at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
       at java.lang.Thread.run(Thread.java:595)
      
      "RMI TCP Accept-0" daemon prio=10 tid=0x00fc5ba8 nid=0x3b runnable [0x5feff000..0x5feffaf0]
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
       - locked <0x78d5bf98> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:450)
       at java.net.ServerSocket.accept(ServerSocket.java:421)
       at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
       at java.lang.Thread.run(Thread.java:595)
      
      "RMI TCP Accept-38688" daemon prio=10 tid=0x00d5e2d0 nid=0x3a runnable [0x5ffff000..0x5ffffb70]
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
       - locked <0x78d5b090> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:450)
       at java.net.ServerSocket.accept(ServerSocket.java:421)
       at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
       at java.lang.Thread.run(Thread.java:595)
      
      "Thread-10" prio=10 tid=0x010ef658 nid=0x37 in Object.wait() [0x601ff000..0x601ff8f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78953ff0> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive)
       at java.lang.Object.wait(Object.java:474)
       at com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive.run(Util.java:857)
       - locked <0x78953ff0> (a com.sun.corba.ee.impl.javax.rmi.CORBA.KeepAlive)
      
      "p: thread-pool-1; w: 2" daemon prio=10 tid=0x01107470 nid=0x36 runnable [0x602ff000..0x602ffbf0]
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
       - locked <0x78d381f8> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:450)
       at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
       at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:250)
       at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107)
       at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
      
      "p: thread-pool-1; w: 1" daemon prio=10 tid=0x00ca0610 nid=0x35 runnable [0x603ff000..0x603ff970]
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
       - locked <0x78d382a8> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:450)
       at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
       at com.sun.corba.ee.impl.transport.SocketOrChannelAcceptorImpl.accept(SocketOrChannelAcceptorImpl.java:250)
       at com.sun.corba.ee.impl.transport.ListenerThreadImpl.doWork(ListenerThreadImpl.java:107)
       at com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:555)
      
      "SelectorThread" daemon prio=10 tid=0x00b8a160 nid=0x34 runnable [0x604ff000..0x604ffc70]
       at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
       at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:164)
       at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:68)
       at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
       - locked <0x78d386f0> (a sun.nio.ch.Util$1)
       - locked <0x78d386e0> (a java.util.Collections$UnmodifiableSet)
       - locked <0x78d384a8> (a sun.nio.ch.DevPollSelectorImpl)
       at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
       at com.sun.corba.ee.impl.transport.SelectorImpl.run(SelectorImpl.java:283)
      
      "Timer-3" daemon prio=10 tid=0x00bb7620 nid=0x33 in Object.wait() [0x605ff000..0x605ffaf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x793ffbc8> (a java.util.TaskQueue)
       at java.lang.Object.wait(Object.java:474)
       at java.util.TimerThread.mainLoop(Timer.java:483)
       - locked <0x793ffbc8> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "Timer-2" prio=10 tid=0x00b627a0 nid=0x32 in Object.wait() [0x606ff000..0x606ffb70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x793ffc50> (a java.util.TaskQueue)
       at java.util.TimerThread.mainLoop(Timer.java:509)
       - locked <0x793ffc50> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "Thread-6" prio=10 tid=0x00cf5808 nid=0x31 waiting on condition [0x607ff000..0x607ff9f0]
       at java.lang.Thread.sleep(Native Method)
       at com.sun.enterprise.admin.server.core.channel.RMIClient.run(RMIClient.java:151)
       at java.lang.Thread.run(Thread.java:595)
      
      "RMI RenewClean-[172.16.1.90:33598,com.sun.enterprise.admin.server.core.channel.LocalRMIClientSocketFactory@1d2bb9f]" daemon prio=10 tid=0x00b6d030 nid=0x2e in Object.wait() [0x60aff000..0x60aff970]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x79400eb0> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
       - locked <0x79400eb0> (a java.lang.ref.ReferenceQueue$Lock)
       at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
       at java.lang.Thread.run(Thread.java:595)
      
      "Timer-1" prio=10 tid=0x006f5f18 nid=0x2d in Object.wait() [0x67b7f000..0x67b7fbf0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x79400368> (a java.util.TaskQueue)
       at java.lang.Object.wait(Object.java:474)
       at java.util.TimerThread.mainLoop(Timer.java:483)
       - locked <0x79400368> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "YJPAgent-RequestListener" daemon prio=1 tid=0x00d56ff8 nid=0x2c runnable [0x6107f000..0x6107fc70]
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
       - locked <0x79400840> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:450)
       at java.net.ServerSocket.accept(ServerSocket.java:421)
       at com.yourkit.runtime.Core$4.run(Core.java:518)
       at java.lang.Thread.run(Thread.java:595)
      
      "YJPAgent-Telemetry" daemon prio=10 tid=0x0077c5a8 nid=0x2b waiting on condition [0x6127f000..0x6127faf0]
       at java.lang.Thread.sleep(Native Method)
       at com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:284)
      
      "RMI LeaseChecker" daemon prio=10 tid=0x00d57568 nid=0x2a waiting on condition [0x6137f000..0x6137fb70]
       at java.lang.Thread.sleep(Native Method)
       at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
       at java.lang.Thread.run(Thread.java:595)
      
      "GC Daemon" daemon prio=2 tid=0x00a3d418 nid=0x28 in Object.wait() [0x6157f000..0x6157fa70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x784072b8> (a sun.misc.GC$LatencyLock)
       at sun.misc.GC$Daemon.run(GC.java:100)
       - locked <0x784072b8> (a sun.misc.GC$LatencyLock)
      
      "RMI Reaper" prio=10 tid=0x00a3d9b0 nid=0x27 in Object.wait() [0x6167f000..0x6167f8f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x7840b1d8> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
       - locked <0x7840b1d8> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
       at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
       at java.lang.Thread.run(Thread.java:595)
      
      "Timer-0" daemon prio=10 tid=0x00744d20 nid=0x26 in Object.wait() [0x6177f000..0x6177f970]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78400230> (a java.util.TaskQueue)
       at java.lang.Object.wait(Object.java:474)
       at java.util.TimerThread.mainLoop(Timer.java:483)
       - locked <0x78400230> (a java.util.TaskQueue)
       at java.util.TimerThread.run(Timer.java:462)
      
      "RMI TCP Accept-0" daemon prio=10 tid=0x00e13228 nid=0x25 runnable [0x6187f000..0x6187fbf0]
       at java.net.PlainSocketImpl.socketAccept(Native Method)
       at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
       - locked <0x78477628> (a java.net.SocksSocketImpl)
       at java.net.ServerSocket.implAccept(ServerSocket.java:450)
       at java.net.ServerSocket.accept(ServerSocket.java:421)
       at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
       at java.lang.Thread.run(Thread.java:595)
      
      "Low Memory Detector" daemon prio=10 tid=0x0017d4c0 nid=0x10 runnable [0x00000000..0x00000000]
      
      "CompilerThread1" daemon prio=10 tid=0x00217640 nid=0xf waiting on condition [0x00000000..0x6757e9a4]
      
      "CompilerThread0" daemon prio=10 tid=0x001f3288 nid=0xe waiting on condition [0x00000000..0x6767ea2c]
      
      "AdapterThread" daemon prio=10 tid=0x00225b38 nid=0xd waiting on condition [0x00000000..0x00000000]
      
      "Signal Dispatcher" daemon prio=10 tid=0x00212ed8 nid=0xc waiting on condition [0x00000000..0x00000000]
      
      "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00220290 nid=0xb runnable [0x00000000..0x00000000]
      
      "YJPAgent-CPUSampler" daemon prio=10 tid=0x00223290 nid=0xa runnable [0x00000000..0x00000000]
      
      "Finalizer" daemon prio=10 tid=0x001974a8 nid=0x8 in Object.wait() [0x6817f000..0x6817fa70]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78407560> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
       - locked <0x78407560> (a java.lang.ref.ReferenceQueue$Lock)
       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
      
      "Reference Handler" daemon prio=10 tid=0x00176870 nid=0x7 in Object.wait() [0x6827f000..0x6827f8f0]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x78403598> (a java.lang.ref.Reference$Lock)
       at java.lang.Object.wait(Object.java:474)
       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
       - locked <0x78403598> (a java.lang.ref.Reference$Lock)
      
      "VM Thread" prio=10 tid=0x001739c0 nid=0x6 runnable
      
      "GC task thread#0 (ParallelGC)" prio=10 tid=0x00137388 nid=0x2 runnable
      
      "GC task thread#1 (ParallelGC)" prio=10 tid=0x00137dc8 nid=0x3 runnable
      
      "GC task thread#2 (ParallelGC)" prio=10 tid=0x00139010 nid=0x4 runnable
      
      "GC task thread#3 (ParallelGC)" prio=10 tid=0x00139a50 nid=0x5 runnable
      
      "VM Periodic Task Thread" prio=10 tid=0x0012ea80 nid=0x11 runnable
      
      



      Thanks,

      Ravi Kiran Bhaskar