5 Replies Latest reply: Oct 17, 2010 4:00 AM by Anuj Khanna RSS

    cpu 100% after particulary request occurs

    Alex Sun Newbie

      i can not reproduce this case, but i have found more than 5 times at least.
      when this case occurs, i analysis the javacore files, and monitor the top command's thread result on my Linux box, then i found below thread is eating cpu.


      And my enviroment :


      WAS6.1, Seam 2.0.2.SP1, JBoss PortletBridge 1.0.0.B3, Richfaces 3.2.1.GA




      the thread details:


      3XMTHREADINFO      "WebContainer : 12" (TID:0x0A591500, sys_thread_t:0x0A198E14, state:CW, native ID:0x00002AFF) prio=5
      4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.fullGetFirstQueuedThread(AbstractQueuedSynchronizer.java:1285)
      4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.getFirstQueuedThread(AbstractQueuedSynchronizer.java:1248)
      4XESTACKTRACE          at java/util/concurrent/locks/ReentrantLock$FairSync.tryAcquire(ReentrantLock.java:223)
      4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:777)
      4XESTACKTRACE          at java/util/concurrent/locks/AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1102)
      4XESTACKTRACE          at java/util/concurrent/locks/ReentrantLock.tryLock(ReentrantLock.java:431)
      4XESTACKTRACE          at org/jboss/seam/core/ConversationEntry.lock(ConversationEntry.java:204)
      4XESTACKTRACE          at org/jboss/seam/core/Manager.restoreAndLockConversation(Manager.java:490)
      4XESTACKTRACE          at org/jboss/seam/core/Manager.restoreConversation(Manager.java:485)
      4XESTACKTRACE          at org/jboss/seam/jsf/SeamPhaseListener.afterRestoreView(SeamPhaseListener.java:388)
      4XESTACKTRACE          at org/jboss/seam/jsf/SeamPhaseListener.afterServletPhase(SeamPhaseListener.java:226(Compiled Code))
      4XESTACKTRACE          at org/jboss/seam/jsf/SeamPhaseListener.afterPhase(SeamPhaseListener.java:192(Compiled Code))
      4XESTACKTRACE          at org/jboss/seam/jsf/CmstarSeamPhaseListenerWrapper.afterPhase(CmstarSeamPhaseListenerWrapper.java:84(Compiled Code))
      4XESTACKTRACE          at org/jboss/portletbridge/lifecycle/LifecyclePhase.execute(LifecyclePhase.java:85(Compiled Code))
      4XESTACKTRACE          at org/jboss/portletbridge/lifecycle/RestoreViewPhase.execute(RestoreViewPhase.java:62)
      4XESTACKTRACE          at org/jboss/portletbridge/lifecycle/PortletLifecycle.execute(PortletLifecycle.java:114)
      4XESTACKTRACE          at javax/faces/webapp/FacesServlet.service(FacesServlet.java:244)
      4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1068(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/ServletWrapper.service(ServletWrapper.java:1009(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:145(Compiled Code))
      4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
      4XESTACKTRACE          at com/wiscom/epstar/web/TimingFilter.doFilter(TimingFilter.java:34)
      4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
      4XESTACKTRACE          at org/jboss/seam/web/RedirectFilter.doFilter(RedirectFilter.java:45)
      4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
      4XESTACKTRACE          at org/jboss/seam/web/ExceptionFilter.doFilter(ExceptionFilter.java:64)
      4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
      4XESTACKTRACE          at org/jboss/seam/web/LoggingFilter.doFilter(LoggingFilter.java:58)
      4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
      4XESTACKTRACE          at org/jboss/seam/servlet/SeamFilter.doFilter(SeamFilter.java:158)
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:130(Compiled Code))
      4XESTACKTRACE          at org/ajax4jsf/webapp/BaseXMLFilter.doXmlFilter(BaseXMLFilter.java:154)
      4XESTACKTRACE          at org/ajax4jsf/webapp/BaseFilter.handleRequest(BaseFilter.java:260)
      4XESTACKTRACE          at org/ajax4jsf/webapp/BaseFilter.processUploadsAndHandleRequest(BaseFilter.java:366)
      4XESTACKTRACE          at org/ajax4jsf/webapp/BaseFilter.doFilter(BaseFilter.java:493)
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain.doFilter(WebAppFilterChain.java:130(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterChain._doFilter(WebAppFilterChain.java:87(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:771(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/filter/WebAppFilterManager.doFilter(WebAppFilterManager.java:679(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:278(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/wswebcontainer/servlet/ServletWrapper.handleRequest(ServletWrapper.java:478(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/servlet/CacheServletWrapper.handleRequest(CacheServletWrapper.java:90(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/WebContainer.handleRequest(WebContainer.java:744(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/wswebcontainer/WebContainer.handleRequest(WebContainer.java:1455(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/webcontainer/channel/WCChannelLink.ready(WCChannelLink.java:115(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleDiscrimination(HttpInboundLink.java:454(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/http/channel/inbound/impl/HttpInboundLink.handleNewInformation(HttpInboundLink.java:383(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/http/channel/inbound/impl/HttpICLReadCallback.complete(HttpICLReadCallback.java:102(Compiled Code))
      4XESTACKTRACE          at com/ibm/ws/tcp/channel/impl/AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165(Compiled Code))
      4XESTACKTRACE          at com/ibm/io/async/AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217(Compiled Code))
      4XESTACKTRACE          at com/ibm/io/async/AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:143(Compiled Code))
      4XESTACKTRACE          at com/ibm/io/async/AsyncFuture.completed(AsyncFuture.java:136(Compiled Code))
      4XESTACKTRACE          at com/ibm/io/async/ResultHandler.complete(ResultHandler.java:195(Compiled Code))
      4XESTACKTRACE          at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:743(Compiled Code))
      4XESTACKTRACE          at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:873)
      4XESTACKTRACE          at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1473(Compiled Code))



        • 1. Re: cpu 100% after particulary request occurs
          Alex Sun Newbie

          javacore file header info:



          NULL           ------------------------------------------------------------------------
          0SECTION       TITLE subcomponent dump routine
          NULL           ===============================
          1TISIGINFO     Dump Event "user" (00004000) received 
          1TIDATETIME    Date:                 2009/03/25 at 09:23:44
          1TIFILENAME    Javacore filename:    /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/javacore.20090325.092344.10374.txt
          NULL           ------------------------------------------------------------------------
          0SECTION       GPINFO subcomponent dump routine
          NULL           ================================
          2XHOSLEVEL     OS Level         : Linux 2.6.9-55.ELsmp
          2XHCPUS        Processors -
          3XHCPUARCH       Architecture   : x86
          3XHNUMCPUS       How Many       : 2
          NULL           
          1XHERROR2      Register dump section only produced for SIGSEGV, SIGILL or SIGFPE.
          NULL           
          NULL           ------------------------------------------------------------------------
          0SECTION       ENVINFO subcomponent dump routine
          NULL           =================================
          1CIJAVAVERSION J2RE 5.0 IBM J9 2.3 Linux x86-32 build j9vmxi3223-20060504
          1CIVMVERSION   VM build 20060501_06428_lHdSMR
          1CIJITVERSION  JIT enabled - 20060428_1800_r8
          1CIRUNNINGAS   Running as a standalone JVM
          1CICMDLINE     /opt/IBM/WebSphere/AppServer/java/bin/java -Declipse.security -Dwas.status.socket=28387 -Dosgi.install.area=/opt/IBM/WebSphere/AppServer -Dosgi.configuration.area=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/configuration -Dosgi.framework.extensions=com.ibm.cds -Xshareclasses:name=webspherev61_%g,groupAccess,nonFatal -Xscmx50M -Xbootclasspath/p:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar -classpath /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/j2ee.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar -Dibm.websphere.internalClassAccessMode=allow -Xms50m -Xmx1024m -Dws.ext.dirs=/opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime -Dderby.system.home=/opt/IBM/WebSphere/AppServer/derby -Dcom.ibm.itp.location=/opt/IBM/WebSphere/AppServer/bin -Djava.util.logging.configureByServer=true -Duser.install.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder -Dwas.install.root=/opt/IBM/WebSphere/AppServer -Dpython.cachedir=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/temp/cachedir -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager -Dserver.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01 -Djava.awt.headless=true;-Dibm.stream.nio=true -Djava.security.auth.login.config=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/wsjaas.conf -Djava.security.policy=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy com.ibm.wsspi.bootstrap.WSPreLauncher -nosplash -application com.ibm.ws.bootstrap.WSLauncher com.ibm.ws.runtime.WsServer /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config idstarCell02 idstarNode03 server1
          1CIJAVAHOMEDIR Java Home Dir:   /opt/IBM/WebSphere/AppServer/java/jre
          1CIJAVADLLDIR  Java DLL Dir:    /opt/IBM/WebSphere/AppServer/java/jre/bin
          1CISYSCP       Sys Classpath:   /opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/vm.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/core.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/charsets.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/graphics.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/security.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmpkcs.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmorb.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmcfw.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmorbapi.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjcefw.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjgssprovider.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjsseprovider2.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmjaaslm.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/ibmcertpathprovider.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/server.jar;/opt/IBM/WebSphere/AppServer/java/jre/lib/xml.jar;
          1CIUSERARGS    UserArgs:
          2CIUSERARG               -Xjcl:jclscar_23
          2CIUSERARG               -Dcom.ibm.oti.vm.bootstrap.library.path=/opt/IBM/WebSphere/AppServer/java/jre/bin
          2CIUSERARG               -Dsun.boot.library.path=/opt/IBM/WebSphere/AppServer/java/jre/bin
          2CIUSERARG               -Djava.library.path=/opt/IBM/WebSphere/AppServer/java/jre/bin:/opt/IBM/WebSphere/AppServer/java/jre/bin/j9vm:/opt/IBM/WebSphere/AppServer/java/jre/bin:/opt/IBM/WebSphere/AppServer/bin:/usr/lib
          2CIUSERARG               -Djava.home=/opt/IBM/WebSphere/AppServer/java/jre
          2CIUSERARG               -Djava.ext.dirs=/opt/IBM/WebSphere/AppServer/java/jre/lib/ext
          2CIUSERARG               -Duser.dir=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
          2CIUSERARG               _j2se_j9
          2CIUSERARG               vfprintf 0x0804B664
          2CIUSERARG               -Declipse.security
          2CIUSERARG               -Dwas.status.socket=28387
          2CIUSERARG               -Dosgi.install.area=/opt/IBM/WebSphere/AppServer
          2CIUSERARG               -Dosgi.configuration.area=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/configuration
          2CIUSERARG               -Dosgi.framework.extensions=com.ibm.cds
          2CIUSERARG               -Xshareclasses:name=webspherev61_%g,groupAccess,nonFatal
          2CIUSERARG               -Xscmx50M
          2CIUSERARG               -Xbootclasspath/p:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmorb.jar:/opt/IBM/WebSphere/AppServer/java/jre/lib/ext/ibmext.jar
          2CIUSERARG               -Dibm.websphere.internalClassAccessMode=allow
          2CIUSERARG               -Xms50m
          2CIUSERARG               -Xmx1024m
          2CIUSERARG               -Dws.ext.dirs=/opt/IBM/WebSphere/AppServer/java/lib:/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/classes:/opt/IBM/WebSphere/AppServer/classes:/opt/IBM/WebSphere/AppServer/lib:/opt/IBM/WebSphere/AppServer/installedChannels:/opt/IBM/WebSphere/AppServer/lib/ext:/opt/IBM/WebSphere/AppServer/web/help:/opt/IBM/WebSphere/AppServer/deploytool/itp/plugins/com.ibm.etools.ejbdeploy/runtime
          2CIUSERARG               -Dderby.system.home=/opt/IBM/WebSphere/AppServer/derby
          2CIUSERARG               -Dcom.ibm.itp.location=/opt/IBM/WebSphere/AppServer/bin
          2CIUSERARG               -Djava.util.logging.configureByServer=true
          2CIUSERARG               -Duser.install.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
          2CIUSERARG               -Djavax.management.builder.initial=com.ibm.ws.management.PlatformMBeanServerBuilder
          2CIUSERARG               -Dwas.install.root=/opt/IBM/WebSphere/AppServer
          2CIUSERARG               -Dpython.cachedir=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/temp/cachedir
          2CIUSERARG               -Djava.util.logging.manager=com.ibm.ws.bootstrap.WsLogManager
          2CIUSERARG               -Dserver.root=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01
          2CIUSERARG               -Djava.awt.headless=true;-Dibm.stream.nio=true
          2CIUSERARG               -Djava.security.auth.login.config=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/wsjaas.conf
          2CIUSERARG               -Djava.security.policy=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/server.policy
          2CIUSERARG               -Dinvokedviajava
          2CIUSERARG               -Djava.class.path=/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties:/opt/IBM/WebSphere/AppServer/properties:/opt/IBM/WebSphere/AppServer/lib/startup.jar:/opt/IBM/WebSphere/AppServer/lib/bootstrap.jar:/opt/IBM/WebSphere/AppServer/lib/j2ee.jar:/opt/IBM/WebSphere/AppServer/lib/lmproxy.jar:/opt/IBM/WebSphere/AppServer/lib/urlprotocols.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batchboot.jar:/opt/IBM/WebSphere/AppServer/deploytool/itp/batch2.jar:/opt/IBM/WebSphere/AppServer/java/lib/tools.jar
          2CIUSERARG               vfprintf
          2CIUSERARG               _port_library 0xF7FFCCA0
          2CIUSERARG               -Xdump



          • 2. Re: cpu 100% after particulary request occurs
            Alex Sun Newbie

            and,



            <core:manager concurrent-request-timeout="1000"
                      conversation-timeout="600000" conversation-id-parameter="cid" />



            • 3. Re: cpu 100% after particulary request occurs
              Stuart Douglas Master

              This looks like an IBM jre bug that I found mentioned here. Maybe upgrade or use a different JVM.

              • 4. Re: cpu 100% after particulary request occurs
                Alex Sun Newbie

                It's a IBM JDK 1.5 bug.

                • 5. Re: cpu 100% after particulary request occurs
                  Anuj Khanna Newbie

                  Hi,


                  We are also facing this issue and need to check if this is a JDK bug, which patch was downloaded and installed to fix this. From where can i download the patch.


                  Also wanted to check if the scenario to reproduce has any similarity to yours


                  If i load and page and while it is loading if i perform another action for example clicking a button or another seam link then CPU usage shoots up.


                  Any help will be highly appreciated.