6 Replies Latest reply on Aug 25, 2014 4:54 PM by Elias Ross

    4.12 high CPU usage in hibernate thread (java.util.regex.Pattern$Start.match)

    Stian Lund Expert

      Hi,

      since 4.12 installation I've noticed a steady increase in CPU usage for RHQ. At appears threads started by Hibernate doing some regex matching is locked and will not finish.

       

      This is the stack trace of the threads:

       

      "HttpManagementService-threads - 529" prio=10 tid=0x00007faee836e000 nid=0x2e32 runnable [0x00007faed63de000]
         java.lang.Thread.State: RUNNABLE
          at java.util.regex.Pattern$Start.match(Pattern.java:3408)
          at java.util.regex.Matcher.search(Matcher.java:1199)
          at java.util.regex.Matcher.find(Matcher.java:592)
          at java.util.regex.Matcher.replaceAll(Matcher.java:902)
          at java.lang.String.replace(String.java:2180)
          at org.jboss.as.jpa.hibernate4.management.QueryName.displayable(QueryName.java:174)
          at org.jboss.as.jpa.hibernate4.management.QueryName.<init>(QueryName.java:120)
          at org.jboss.as.jpa.hibernate4.management.QueryName.queryName(QueryName.java:110)
          at org.jboss.as.jpa.hibernate4.management.HibernateStatisticsResource.getQueryNames(HibernateStatisticsResource.java:345)
          at org.jboss.as.jpa.hibernate4.management.HibernateStatisticsResource.hasQuery(HibernateStatisticsResource.java:332)
          at org.jboss.as.jpa.hibernate4.management.HibernateStatisticsResource.hasChild(HibernateStatisticsResource.java:81)
          at org.jboss.as.controller.OperationContextImpl.getAuthorizationResource(OperationContextImpl.java:1171)
          at org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(OperationContextImpl.java:1129)
          at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1052)
          at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:848)
          at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecuteInternal(ReadResourceHandler.java:265)
          at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecute(ReadResourceHandler.java:142)
          at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:249)
          at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:601)
          at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:479)
          at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:283)
          at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:278)
          at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:230)
          at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:136)
          at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:436)
          at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:426)
          at org.jboss.as.domain.http.server.DomainApiHandler.processRequest(DomainApiHandler.java:263)
          at org.jboss.as.domain.http.server.DomainApiHandler.doHandle(DomainApiHandler.java:167)
          at org.jboss.as.domain.http.server.DomainApiHandler.handle(DomainApiHandler.java:174)
          at org.jboss.as.domain.http.server.security.SubjectAssociationHandler$1.run(SubjectAssociationHandler.java:69)
          at org.jboss.as.domain.http.server.security.SubjectAssociationHandler$1.run(SubjectAssociationHandler.java:65)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
          at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handleRequest(SubjectAssociationHandler.java:65)
          at org.jboss.as.domain.http.server.security.SubjectAssociationHandler.handle(SubjectAssociationHandler.java:59)
          at org.jboss.as.domain.http.server.DomainApiCheckHandler.handle(DomainApiCheckHandler.java:45)
          at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
          at org.jboss.sun.net.httpserver.AuthFilter.doFilter(AuthFilter.java:69)
          at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
          at org.jboss.sun.net.httpserver.ServerImpl$Exchange$LinkHandler.handle(ServerImpl.java:710)
          at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:78)
          at org.jboss.as.domain.http.server.RealmReadinessFilter.doFilter(RealmReadinessFilter.java:47)
          at org.jboss.as.domain.http.server.DmrFailureReadinessFilter.doFilter(DmrFailureReadinessFilter.java:45)
          at org.jboss.com.sun.net.httpserver.Filter$Chain.doFilter(Filter.java:81)
          at org.jboss.sun.net.httpserver.ServerImpl$Exchange.run(ServerImpl.java:682)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          at java.lang.Thread.run(Thread.java:744)
          at org.jboss.threads.JBossThread.run(JBossThread.java:122)
      
      

       

      This is affecting both or Test RHQ installations, running RHEL 6.5 with Java 1.7_45. They are dual CPU VMs and currently two threads are running at 100% causing almost full load.

       

      The CPU is slowly building over time, see this graph of user load:

       

      RHQ-Cpu1.png

       

      Any ideas what might cause this?

       

      Stian