6 Replies Latest reply on Jan 8, 2019 8:43 PM by Jim Fan

    JBoss EAP 6.2.3 Hangs with AJP Threads Blocked

    Jim Fan Newbie

      Dear all,

       

      My first post on JBoss Developer, I would try to state the problem as accurate as I can.

       

      Setup: 2 x load balanced IIS on Windows Server 2012, forwarding request to 2 x JBoss (on same Windows Servers) in cluster with session replication

       

      Fault symptom: From end-user perspective, website becomes unresponsive. Service only resumes after restarting both JBoss instances manually.

       

      The fault underwent a very lengthy troubleshooting process and allow me to skip ahead to the last two steps performed:

       

      Firstly, at incident time, thread dump was performed using jboss-cli.bat comes with JBoss. Findings:

       

      • Many AJP threads blocked by older AJP threads:

              {

                  "thread-id" => 991L,

                  "thread-name" => "ajp-{server name}-cluster/{IP address}:8009-175",

                  "thread-state" => "BLOCKED",

                  "blocked-time" => -1L,

                  "blocked-count" => 1L,

                  "waited-time" => -1L,

                  "waited-count" => 0L,

                  "lock-info" => {

                      "class-name" => "org.jboss.as.web.session.SessionBasedClusteredSession",

                      "identity-hash-code" => 1445846157

                  },

                  "lock-name" => "org.jboss.as.web.session.SessionBasedClusteredSession@562ddc8d",

                  "lock-owner-id" => 855L,

                  "lock-owner-name" => "ajp-{server name}-cluster/{IP address}:8009-80",

                  "stack-trace" => [

                      {

                          "file-name" => "DistributableSessionManager.java",

                          "line-number" => 858,

                          "class-name" => "org.jboss.as.web.session.DistributableSessionManager",

                          "method-name" => "storeSession",

                          "native-method" => false

                      },

                      {

                          "file-name" => "InstantSnapshotManager.java",

                          "line-number" => 47,

                          "class-name" => "org.jboss.as.web.session.InstantSnapshotManager",

                          "method-name" => "snapshot",

                          "native-method" => false

                      },

                      {

                          "file-name" => "ClusteredSessionValve.java",

                          "line-number" => 142,

                          "class-name" => "org.jboss.as.web.session.ClusteredSessionValve",

                          "method-name" => "handleRequest",

                          "native-method" => false

                      },

                      {

                          "file-name" => "ClusteredSessionValve.java",

                          "line-number" => 99,

                          "class-name" => "org.jboss.as.web.session.ClusteredSessionValve",

                          "method-name" => "invoke",

                          "native-method" => false

                      },

                      {

                          "file-name" => "JvmRouteValve.java",

                          "line-number" => 92,

                          "class-name" => "org.jboss.as.web.session.JvmRouteValve",

                          "method-name" => "invoke",

                          "native-method" => false

                      },

                      {

                          "file-name" => "LockingValve.java",

                          "line-number" => 64,

                          "class-name" => "org.jboss.as.web.session.LockingValve",

                          "method-name" => "invoke",

                          "native-method" => false

                      },

                (stacktrace skipped)

       

       

      • The blockers are found waiting for FlowControl credit:

              {

                  "thread-id" => 855L,

                  "thread-name" => "ajp-{server name}-cluster/{IP address}:8009-80",

                  "thread-state" => "TIMED_WAITING",

                  "blocked-time" => -1L,

                  "blocked-count" => 1L,

                  "waited-time" => -1L,

                  "waited-count" => 182L,

                  "lock-info" => {

                      "class-name" => "org.jgroups.protocols.FlowControl$Credit",

                      "identity-hash-code" => 1691984096

                  },

                  "lock-name" => "org.jgroups.protocols.FlowControl$Credit@64d9a0e0",

                  "lock-owner-id" => -1L,

                  "lock-owner-name" => undefined,

                  "stack-trace" => [

                      {

                          "file-name" => "Object.java",

                          "line-number" => -2,

                          "class-name" => "java.lang.Object",

                          "method-name" => "wait",

                          "native-method" => true

                      },

                      {

                          "file-name" => "FlowControl.java",

                          "line-number" => 553,

                          "class-name" => "org.jgroups.protocols.FlowControl$Credit",

                          "method-name" => "decrementIfEnoughCredits",

                          "native-method" => false

                      },

                      {

                          "file-name" => "UFC.java",

                          "line-number" => 114,

                          "class-name" => "org.jgroups.protocols.UFC",

                          "method-name" => "handleDownMessage",

                          "native-method" => false

                      },

                      {

                          "file-name" => "FlowControl.java",

                          "line-number" => 341,

                          "class-name" => "org.jgroups.protocols.FlowControl",

                          "method-name" => "down",

                          "native-method" => false

                      },

                      {

                          "file-name" => "FlowControl.java",

                          "line-number" => 351,

                          "class-name" => "org.jgroups.protocols.FlowControl",

                          "method-name" => "down",

                          "native-method" => false

                      },

                      {

                          "file-name" => "FRAG2.java",

                          "line-number" => 247,

                          "class-name" => "org.jgroups.protocols.FRAG2",

                          "method-name" => "fragment",

                          "native-method" => false

                      },

                      {

                          "file-name" => "FRAG2.java",

                          "line-number" => 131,

                          "class-name" => "org.jgroups.protocols.FRAG2",

                          "method-name" => "down",

                          "native-method" => false

                      },

                      {

                          "file-name" => "RSVP.java",

                          "line-number" => 143,

                          "class-name" => "org.jgroups.protocols.RSVP",

                          "method-name" => "down",

                          "native-method" => false

                      },

                      {

                          "file-name" => "ProtocolStack.java",

                          "line-number" => 1030,

                          "class-name" => "org.jgroups.stack.ProtocolStack",

                          "method-name" => "down",

                          "native-method" => false

                      },

                      {

                          "file-name" => "JChannel.java",

                          "line-number" => 722,

                          "class-name" => "org.jgroups.JChannel",

                          "method-name" => "down",

                          "native-method" => false

                      },

                     (stackstrace skipped)

       

       

      After consulting documentation, it is found JGroups is the credit-based protocol used for session replication. We then tried to workaround by removing "<distributable />" from web.xml of web application deployed to both JBoss. The problem does not happen since then.

       

      Despite this we would still very much like to find out the root cause. After all, if it is caused by application code, the pitfall should be avoided at all cost. Or could it be a bug in JGroups or even JBoss? This is what I can't rule out as well.

       

      Appreciate any input.

       

      Thanks,

      Jim

        • 1. Re: JBoss AS 7.3.3 Hangs with AJP Threads Blocked
          Jim Fan Newbie

          Hi all,

           

          Thanks to those who have viewed this thread.

           

          Supplementary info: It was once suggested that if HTTP session is too large in size, JGroups could have FlowControl credit exhausted. For this I have made use of Servlet filter from Redhat to serialise then print out HTTP session attributes and their size:

           

          2018-11-08 15:36:05.850 INFO  [pool-9-thread-31] (StartupService.java:42) - Init cache

          2018-11-08 15:36:08.616 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute isPriceChanged = 82 bytes

          2018-11-08 15:36:08.616 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute org.springframework.web.context.request.ServletRequestAttributes.DESTRUCTION_CALLBACK.scopedTarget.couponValidationService = 3,351 bytes

          2018-11-08 15:36:08.631 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute basket = 382,760 bytes

          2018-11-08 15:36:08.631 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute scopedTarget.couponValidationService = 5 bytes

          2018-11-08 15:36:08.631 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute onlineBasket = 5 bytes

          2018-11-08 15:36:08.631 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute javax.servlet.jsp.jstl.fmt.request.charset = 8 bytes

          2018-11-08 15:36:08.631 INFO  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:76) - HttpSession xNakzZU1+droXhToW4UPO3BW = 386,211 bytes

           

           

           

          Which could hit a larger size if more items are put into the shopping cart. But this is likely to be the largest session according to real-life business logic:

           

          2018-11-08 15:36:43.828 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute isPriceChanged = 82 bytes

          2018-11-08 15:36:43.828 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute org.springframework.web.context.request.ServletRequestAttributes.DESTRUCTION_CALLBACK.scopedTarget.couponValidationService = 3,351 bytes

          2018-11-08 15:36:43.828 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute basket = 383,750 bytes

          2018-11-08 15:36:43.828 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute scopedTarget.couponValidationService = 5 bytes

          2018-11-08 15:36:43.828 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute onlineBasket = 161,333 bytes

          2018-11-08 15:36:43.828 FINE  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:56) - HttpSession attribute javax.servlet.jsp.jstl.fmt.request.charset = 8 bytes

          2018-11-08 15:36:43.828 INFO  [http-/0.0.0.0:8081-7] (HttpSessionSizeFilter.java:76) - HttpSession xNakzZU1+droXhToW4UPO3BW = 548,529 bytes

           

           

          So a large HTTP session could be as large as 535.67Kb. But I have seen a lot bigger ones from other developers. And it does not seem to be the root cause. I post it for reference anyway.

           

          Thanks,

          Jim

          • 2. Re: JBoss AS 7.3.3 Hangs with AJP Threads Blocked
            Jim Fan Newbie

            Hi all,

             

            I guess it is because JBoss AS 7.3.3 is rather dated and incentive in viewing / answering is low. Anyway, I've still been searching on the topic and found my error resembles two JBoss JIRA cases, in that threads wait indefinitely in FlowControl.decrementIfEnoughCredits method. Share them here in case anyone will find them useful:

             

            [JGRP-1675] Threads stuck in FlowControl.decrementIfEnoughCredits - JBoss Issue Tracker

             

            [ISPN-3645] StateTransferLargeObjectTest hangs randomly - JBoss Issue Tracker

             

            Jim

            • 3. Re: JBoss AS 7.3.3 Hangs with AJP Threads Blocked
              Jaikiran Pai Master

              >> My first post on JBoss Developer, I would try to state the problem as accurate as I can.

               

              Welcome to the forums, Jim.

               

              >> I guess it is because JBoss AS 7.3.3 is rather dated and incentive in viewing / answering is low.

               

              Your descriptions about the issue have been detailed enough, so thank you for that. However, it's not clear which version of JBoss AS you are using. There isnt' a JBoss AS (community edition) 7.3.3. Are you maybe using JBoss EAP (the enterprise supported edition)?

               

              >> Anyway, I've still been searching on the topic and found my error resembles two JBoss JIRA cases, in that threads wait indefinitely in FlowControl.decrementIfEnoughCredits method.

               

              Based on that, it looks like this issue is more related to session replication backed by Infinispan? Would it be possible to get a more detailed thread dump from the JVM where this is actually blocked? I see that you did paste something that resembles a thread dump in the first post, but the actually thread dumps typically are much more (verbose and) useful. Can you get those and attach to this thread? Maybe that will have hints on what the issue is. There are multiple ways to get a thread dump, but typically the jstack command (that's part of the JDK installation in JAVA_HOME/bin) against the JBoss AS JVM should work.

              • 4. Re: JBoss AS 7.3.3 Hangs with AJP Threads Blocked
                Jim Fan Newbie

                Hello jaikiran,

                 

                Thank you so much for your feedback. JBoss version is indeed EAP 6.2.3.GA

                 

                More of server.log:

                 

                07:39:40,469 INFO  [org.jboss.modules] (main) JBoss Modules version 1.3.3.Final-redhat-1
                07:39:40,703 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA-redhat-1
                07:39:40,828 INFO  [org.jboss.as] (MSC service thread 1-6) JBAS015899: JBoss EAP 6.2.3.GA (AS 7.3.3.Final-redhat-3) starting
                07:39:40,828 DEBUG [org.jboss.as.config] (MSC service thread 1-6) Configured system properties:
                awt.toolkit = sun.awt.windows.WToolkit
                file.encoding = Cp1252
                file.encoding.pkg = sun.io
                file.separator = \
                java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
                java.awt.printerjob = sun.awt.windows.WPrinterJob
                java.class.path = D:\jboss\jboss-eap-6.2\jboss-modules.jar
                java.class.version = 51.0
                java.endorsed.dirs = C:\Program Files\Java\jre7\lib\endorsed
                java.ext.dirs = C:\Program Files\Java\jre7\lib\ext;C:\Windows\Sun\Java\lib\ext
                java.home = C:\Program Files\Java\jre7
                java.io.tmpdir = C:\Users\SERVER-NAME-MASKEDSE~1\AppData\Local\Temp\
                java.library.path = C:\Windows\system32;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\PROGRA~2\BMCSOF~1\common\globalc\bin\Windows-x86;C:\PROGRA~2\BMCSOF~1\common\globalc\bin\Windows-x86-64;C:\Program Files\Microsoft Network Monitor 3\;C:\PROGRA~2\BMCSOF~1\Patrol3\bin;C:\Program Files (x86)\BMC Software\Patrol3\bin\patrol_x86;.
                java.net.preferIPv4Stack = true
                java.runtime.name = Java(TM) SE Runtime Environment
                java.runtime.version = 1.7.0_60-b19
                java.specification.name = Java Platform API Specification
                java.specification.vendor = Oracle Corporation
                java.specification.version = 1.7
                java.util.logging.manager = org.jboss.logmanager.LogManager
                java.vendor = Oracle Corporation
                java.vendor.url = http://java.oracle.com/
                java.vendor.url.bug = http://bugreport.sun.com/bugreport/
                java.version = 1.7.0_60
                java.vm.info = mixed mode
                java.vm.name = Java HotSpot(TM) 64-Bit Server VM
                java.vm.specification.name = Java Virtual Machine Specification
                java.vm.specification.vendor = Oracle Corporation
                java.vm.specification.version = 1.7
                java.vm.vendor = Oracle Corporation
                java.vm.version = 24.60-b09
                javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
                javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
                javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
                javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
                javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
                javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
                javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
                javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
                javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
                javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
                jboss.bind.address = SERVER-NAME-MASKED-cluster
                jboss.default.multicast.address = 230.1.1.130
                jboss.home.dir = D:\jboss\jboss-eap-6.2
                jboss.host.name = SERVER-NAME-MASKED
                jboss.modules.dir = D:\jboss\jboss-eap-6.2\modules
                jboss.modules.system.pkgs = org.jboss.byteman
                jboss.node.name = node3
                jboss.qualified.host.name = SERVER-NAME-MASKED
                jboss.server.base.dir = D:\jboss\jboss-eap-6.2\standalone
                jboss.server.config.dir = D:\jboss\jboss-eap-6.2\standalone\configuration
                jboss.server.data.dir = D:\jboss\jboss-eap-6.2\standalone\data
                jboss.server.deploy.dir = D:\jboss\jboss-eap-6.2\standalone\data\content
                jboss.server.log.dir = D:\jboss\jboss-eap-6.2\standalone\log
                jboss.server.name = SERVER-NAME-MASKED
                jboss.server.persist.config = true
                jboss.server.temp.dir = D:\jboss\jboss-eap-6.2\standalone\tmp
                line.separator = 
                
                logging.configuration = file:D:\jboss\jboss-eap-6.2\standalone/configuration/logging.properties
                module.path = D:\jboss\jboss-eap-6.2\modules
                org.jboss.boot.log.file = D:\jboss\jboss-eap-6.2\standalone\log\server.log
                org.jboss.resolver.warning = true
                org.xml.sax.driver = __redirected.__XMLReaderFactory
                os.arch = amd64
                os.name = Windows Server 2012 R2
                os.version = 6.3
                path.separator = ;
                program.name = standalone.bat
                sun.arch.data.model = 64
                sun.boot.class.path = C:\Program Files\Java\jre7\lib\resources.jar;C:\Program Files\Java\jre7\lib\rt.jar;C:\Program Files\Java\jre7\lib\sunrsasign.jar;C:\Program Files\Java\jre7\lib\jsse.jar;C:\Program Files\Java\jre7\lib\jce.jar;C:\Program Files\Java\jre7\lib\charsets.jar;C:\Program Files\Java\jre7\lib\jfr.jar;C:\Program Files\Java\jre7\classes
                sun.boot.library.path = C:\Program Files\Java\jre7\bin
                sun.cpu.endian = little
                sun.cpu.isalist = amd64
                sun.desktop = windows
                sun.io.unicode.encoding = UnicodeLittle
                sun.java.command = D:\jboss\jboss-eap-6.2\jboss-modules.jar -mp D:\jboss\jboss-eap-6.2\modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=D:\jboss\jboss-eap-6.2 --server-config=standalone-ha.xml -Djboss.node.name=node3 -b SERVER-NAME-MASKED-cluster -Djava.net.preferIPv4Stack=true -u 230.1.1.130
                sun.java.launcher = SUN_STANDARD
                sun.jnu.encoding = Cp1252
                sun.management.compiler = HotSpot 64-Bit Tiered Compilers
                sun.os.patch.level = 
                user.country = GB
                user.dir = D:\jboss\jboss-eap-6.2\bin
                user.home = C:\Users\SERVER-NAME-MASKEDService
                user.language = en
                user.name = SERVER-NAME-MASKEDService
                user.script = 
                user.timezone = Asia/Shanghai
                user.variant = 
                07:39:40,828 DEBUG [org.jboss.as.config] (MSC service thread 1-6) VM Arguments: -XX:+UseCompressedOops -Dprogram.name=standalone.bat -Xms2G -Xmx4G -XX:MaxPermSize=1G -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.boot.log.file=D:\jboss\jboss-eap-6.2\standalone\log\server.log -Dlogging.configuration=file:D:\jboss\jboss-eap-6.2\standalone/configuration/logging.properties 
                07:39:41,672 DEBUG [org.jgroups.conf.ClassConfigurator] (Controller Boot Thread) Using jg-magic-map.xml as magic number file and jg-protocol-ids.xml for protocol IDs
                07:39:42,297 INFO  [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.9.GA-redhat-1
                07:39:42,297 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
                07:39:42,297 INFO  [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.9.GA-redhat-1
                07:39:42,328 INFO  [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.19.GA-redhat-1
                07:39:42,391 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-3) JBAS010408: Starting JCA Subsystem (IronJacamar 1.0.23.1.Final-redhat-1)
                07:39:42,469 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 46) JBAS013171: Activating Security Subsystem
                07:39:42,485 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Driver (version 1.3)
                07:39:42,485 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 50) JBAS015537: Activating WebServices Extension
                07:39:42,500 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 38) JBAS012605: Activated the following JSF Implementations: [main, 1.2]
                07:39:42,500 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 31) JBAS010280: Activating Infinispan subsystem.
                07:39:42,516 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 41) JBAS011800: Activating Naming Subsystem
                07:39:42,516 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 35) JBAS010260: Activating JGroups subsystem.
                07:39:42,516 INFO  [org.jboss.as.security] (MSC service thread 1-1) JBAS013170: Current PicketBox version=4.0.19.SP6-redhat-1
                07:39:42,688 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) JBAS015400: Bound mail session [java:jboss/mail/Default]
                07:39:42,703 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: Starting Naming Service
                07:39:42,813 INFO  [org.apache.coyote.ajp] (MSC service thread 1-2) JBWEB003046: Starting Coyote AJP/1.3 on ajp-SERVER-NAME-MASKED-cluster/172.19.34.83:8009
                07:39:42,860 INFO  [org.apache.coyote.http11] (MSC service thread 1-6) JBWEB003001: Coyote HTTP/1.1 initializing on : http-SERVER-NAME-MASKED-cluster/172.19.34.83:8080
                07:39:42,860 INFO  [org.apache.coyote.http11] (MSC service thread 1-6) JBWEB003000: Coyote HTTP/1.1 starting on: http-SERVER-NAME-MASKED-cluster/172.19.34.83:8080
                07:39:43,016 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 52) MODCLUSTER000001: Initializing mod_cluster ${project.version}
                07:39:43,032 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 52) MODCLUSTER000032: Listening to proxy advertisements on /224.0.1.105:23364
                07:39:43,157 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-6) JBAS015012: Started FileSystemDeploymentService for directory D:\jboss\jboss-eap-6.2\standalone\deployments
                07:39:43,157 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-1) JBAS015876: Starting deployment of "SERVER-NAME-MASKED-SOMETHING.war" (runtime-name: "SERVER-NAME-MASKED-SOMETHING.war")
                07:39:43,157 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) JBAS015876: Starting deployment of "SERVER-NAME-MASKED-SOMETHING.war" (runtime-name: "SERVER-NAME-MASKED-SOMETHING.war")
                07:39:43,172 INFO  [org.jboss.as.remoting] (MSC service thread 1-6) JBAS017100: Listening on 127.0.0.1:9999
                07:39:43,172 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100: Listening on 172.19.34.83:4447
                07:39:43,266 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
                07:39:43,266 INFO  [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022052: Starting JBoss Web Services - Stack CXF Server 4.2.4.Final-redhat-1
                07:39:45,625 WARN  [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015893: Encountered invalid class name 'org.xmlpull.mxp1.MXParser,org.xmlpull.mxp1_serializer.MXSerializer' for service type 'org.xmlpull.v1.XmlPullParserFactory'
                

                 

                 

                You are right that we are using session replication backed by infinispan. Below is excerpt from standalone-ha.xml :

                 

                        <subsystem xmlns="urn:jboss:domain:infinispan:1.4">
                            <!-- skipped some -->
                            <cache-container name="web" aliases="standard-session-cache" default-cache="repl" module="org.jboss.as.clustering.web.infinispan">
                                <transport lock-timeout="60000"/>
                                <replicated-cache name="repl" mode="ASYNC" batching="true">
                                    <locking isolation="REPEATABLE_READ"/>
                                </replicated-cache>
                                <replicated-cache name="sso" mode="SYNC" batching="true"/>
                                <distributed-cache name="dist" l1-lifespan="0" mode="ASYNC" batching="true">
                                </distributed-cache>
                            </cache-container>
                            <!-- skipped some -->
                        </subsystem>
                

                 

                        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="udp">
                            <stack name="udp">
                                <transport type="UDP" socket-binding="jgroups-udp"/>
                                <protocol type="PING"/>
                                <protocol type="MERGE3"/>
                                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                                <protocol type="FD"/>
                                <protocol type="VERIFY_SUSPECT"/>
                                <protocol type="pbcast.NAKACK"/>
                                <protocol type="UNICAST2"/>
                                <protocol type="pbcast.STABLE"/>
                                <protocol type="pbcast.GMS"/>
                                <protocol type="UFC"/>
                                <protocol type="MFC"/>
                                <protocol type="FRAG2"/>
                                <protocol type="RSVP"/>
                            </stack>
                            <stack name="tcp">
                                <transport type="TCP" socket-binding="jgroups-tcp"/>
                                <protocol type="MPING" socket-binding="jgroups-mping"/>
                                <protocol type="MERGE2"/>
                                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                                <protocol type="FD"/>
                                <protocol type="VERIFY_SUSPECT"/>
                                <protocol type="pbcast.NAKACK"/>
                                <protocol type="UNICAST2"/>
                                <protocol type="pbcast.STABLE"/>
                                <protocol type="pbcast.GMS"/>
                                <protocol type="UFC"/>
                                <protocol type="MFC"/>
                                <protocol type="FRAG2"/>
                                <protocol type="RSVP"/>
                            </stack>
                        </subsystem>
                

                 

                 

                The thread dump posted was generated by using jboss-cli.bat in JBoss /bin directory. We had to do this because security policy forbids me from installing JDK. So jboss-cli.bat is the best tool available.

                 

                Anyway I have the dump files attached.

                 

                Thanks,

                Jim

                • 5. Re: JBoss EAP 6.2.3 Hangs with AJP Threads Blocked
                  Jaikiran Pai Master

                  >> JBoss version is indeed EAP 6.2.3.GA

                   

                  I haven't yet had a chance to look at the attached thread dumps, but since you say you are using JBoss EAP, I just wanted to make a mention that if you file a support issue through the JBoss EAP support channels, chances are you will get help much more quickly than the community forums here.

                  • 6. Re: JBoss EAP 6.2.3 Hangs with AJP Threads Blocked
                    Jim Fan Newbie

                    Hello Jaikiran,

                     

                    That's okay. The log files are indeed very extensive.

                     

                    I actually had filed several support cases to Redhat but I would rather not comment on that experience. Also aware this forum is not for troubleshooting (but discussion, of course), and that's why I didn't upload the log / setting files in the first place.

                     

                    Anyway, thanks a lot for your input and review.

                     

                    (Edited) Guess this is a good start to involve myself in JBoss community. The forum as well as JBoss Issue Tracker are exceptionally informative.

                     

                    Jim