9 Replies Latest reply on Nov 1, 2006 6:44 AM by subramaniam.venkat

    JBoss Hangs even after the implementation of Arjuna Transact

    subramaniam.venkat

      Hello All,

      I had probelms in my application that after some time the application was not responding to any request at that time i was using the older version of JBoss which was JBoss-3.2.4 and i was using the fast in memory transaction manager with this version .

      Then i read the article by Mark Little in the forum

      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=85779

      And i ported my application to the new JBoss-4.0.3SP1 as mentioned and also applied the new Arjuna transaction Manager and for that i followed the installation guide .

      http://labs.jboss.com/file-access/default/members/jbosstm/downloads/4.2.1/notes/jta_install.txt

      But even after that i am not getting out of the hanging situation . After sometime the JBoss goes into a sleep mode from where it is not recovering and it is not answering to any of the request.

      I did a truss at that point and the output is mentioned below.

      Can some one let me know whether i have to do any code adaptation for this.

      Please let me know your comments.

      Thanks & Warm Regards,
      Subramaniam

      poll(0xB1C01588, 0, 10000) = 0
      lwp_cond_wait(0x006D53A0, 0x006D5388, 0xB17014A0) Err#62 ETIME
      poll(0xB1901590, 0, 20000) = 0
      lwp_cond_wait(0x01A7EEA8, 0x01A7EE90, 0xB1481440) Err#62 ETIME
      lwp_cond_wait(0x014E6910, 0x014E68F8, 0xB12815A8) Err#62 ETIME
      poll(0xB0181578, 0, 60000) = 0
      lwp_cond_wait(0x00B71550, 0x00B71538, 0xB2C01518) Err#62 ETIME
      lwp_cond_wait(0x015061F0, 0x015061D8, 0xADB81560) Err#62 ETIME
      lwp_continue(51) = 0
      lwp_continue(45) = 0
      lwp_mutex_wakeup(0x0002CFE0) = 0
      lwp_mutex_lock(0x0002CFE0) = 0
      lwp_mutex_wakeup(0x0002CFE0) = 0
      lwp_mutex_lock(0x0002CFE0) = 0
      lwp_mutex_wakeup(0x0002CFE0) = 0
      lwp_mutex_lock(0x0002CFE0) = 0
      poll(0xAE581560, 0, 120000) = 0
      poll(0xB4A01B18, 0, 50) = 0
      lwp_mutex_wakeup(0x0002CFE0) = 0
      lwp_mutex_lock(0x0002CFE0) = 0
      poll(0xB4A01B18, 0, 50) = 0
      open64("PutObjectStoreDirHere/HashedActionStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction", O_RDONLY|O_NDELAY) = 64

        • 1. Re: JBoss Hangs even after the implementation of Arjuna Tran
          kconner

          You need to find out what is causing the problem first. Please look at http://wiki.jboss.org/wiki/Wiki.jsp?page=StackTrace

          The thread dump should indicate why threads are blocking.

          • 2. Re: JBoss Hangs even after the implementation of Arjuna Tran
            subramaniam.venkat

            Hello

            My enviroment details are

            Solaris, JDK142 and JBoss-4.0.3SP1 with Arjuna Transaction Manager.
            And i am starting the run.sh with java.exe

            Thanks & Warm Regards
            Subramaniam V

            • 3. Re: JBoss Hangs even after the implementation of Arjuna Tran
              kconner

              You need to generate a thread dump and examine it.

              kill -3 <pid>
              will send the thread dump to your console.


              • 4. Re: JBoss Hangs even after the implementation of Arjuna Tran
                subramaniam.venkat

                Hello Conner,

                I am running the run.sh as a background process . When i did the
                following suggested by you

                kill -3 pid

                i got these messages

                2006-10-31 12:33:38,298 INFO [org.jboss.system.server.Server] LifeThread.run exits!
                2006-10-31 12:33:38,299 INFO [org.jboss.system.server.Server] Shutting down the JVM now!
                2006-10-31 12:33:38,301 INFO [org.jboss.system.server.Server] Runtime shutdown hook called, forceHalt: true
                2006-10-31 12:33:38,301 INFO [org.jboss.system.server.Server] JBoss SHUTDOWN: Undeploying all packages

                and then all the EJB's and war files which i had deployed got undeployed.

                But what is thread dump and how should i start it in JBoss.
                Can you please let me know .

                Thanks & Warm Regards,
                Subramaniam V

                • 5. Re: JBoss Hangs even after the implementation of Arjuna Tran
                  subramaniam.venkat

                  And also i got this message at the end of the shutdown

                  2006-10-31 12:36:55,557 ERROR [org.apache.tomcat.util.net.PoolTcpEndpoint] Socket error caused by remote host /10.255.252.13
                  java.net.SocketException: Invalid argument
                  at java.net.PlainSocketImpl.socketSetOption(Native Method)
                  at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:240)
                  at java.net.Socket.setTcpNoDelay(Socket.java:771)
                  at org.apache.tomcat.util.net.PoolTcpEndpoint.setSocketOptions(PoolTcpEndpoint.java:503)
                  at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:515)
                  at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
                  at java.lang.Thread.run(Thread.java:534)
                  2006-10-31 12:36:56,560 INFO [org.jboss.web.tomcat.tc5.StandardService] Stopping service jboss.web
                  2006-10-31 12:36:56,595 INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] undeploy, ctxPath=/invoker, warUrl=.../deploy/http-invoker.sar/invoker.war/
                  2006-10-31 12:36:56,598 INFO [org.jboss.web.tomcat.tc5.TomcatDeployer] undeploy, ctxPath=/jboss-net, warUrl=.../deploy/jboss-net.sar/jboss-net.war/
                  2006-10-31 12:36:57,270 INFO [org.jboss.system.server.Server] Shutdown complete

                  Thanks & warm Regards,
                  Subramaniam V

                  • 6. Re: JBoss Hangs even after the implementation of Arjuna Tran
                    kconner

                    SIGQUIT should not have stopped the VM. Can you please try again using

                    kill -QUIT <pid>



                    • 7. Re: JBoss Hangs even after the implementation of Arjuna Tran
                      subramaniam.venkat

                      Hello Conner,

                      I executed the kill command suggested by you but there is no response from the JBoss ie it is not shutting down.

                      Can you please tell what is use of executing this command to do with the no response state of JBoss.

                      I have also changed jboss-service.xml like

                      <mbean code="com.arjuna.ats.jbossatx.jts.TransactionManagerService"
                      name="jboss:service=TransactionManager"
                      xmbean-dd="resource:xmdesc/TransactionManagerService-xmbean.xml">
                      <attribute name="TransactionTimeout">300</attribute>
                      <attribute name="StatisticsEnabled">true</attribute>
                      <attribute name="RunInVMRecoveryManager">true</attribute>
                      </mbean>


                      And can you also tell what this RunInVMRecoveryManager means.

                      Thanks & Warm Regards,
                      Subramaniam V

                      • 8. Re: JBoss Hangs even after the implementation of Arjuna Tran
                        kconner

                        You should have a thread dump on the jboss console, this should show you the reason for your threads locking up.

                        Please refer to http://wiki.jboss.org/wiki/Wiki.jsp?page=StackTrace

                        • 9. Re: JBoss Hangs even after the implementation of Arjuna Tran
                          subramaniam.venkat

                          Hello Conner,

                          Please find the attached thread dump taken from the jmx-console . The first thread dump is taken when the JBoss is responding to all request which i am sending. And the second one is taken when the response comes very very late . And i also executed the top command at the point when the response was to slow and i saw that the java process was in sleep mode and never came into the run or occupied the cpu.

                          Please let me know what i should do to avoid coming into this situation.



                          Thanks & warm regards,
                          Subramaniam V

                          ========================================
                          Total Threads: 69
                          Total Thread Groups: 9

                          Thread Group: system : max priority:10, demon:false

                          Thread: Reference Handler : priority:10, demon:true
                          Thread: Finalizer : priority:8, demon:true
                          Thread: Signal Dispatcher : priority:10, demon:true
                          Thread: AdapterThread : priority:10, demon:true
                          Thread: CompilerThread0 : priority:10, demon:true
                          Thread: CompilerThread1 : priority:10, demon:true
                          Thread: RMI TCP Accept-0 : priority:5, demon:true
                          Thread: RMI Reaper : priority:5, demon:false
                          Thread: GC Daemon : priority:2, demon:true
                          Thread: RMI TCP Accept-4444 : priority:5, demon:true
                          Thread: RMI TCP Accept-0 : priority:5, demon:true

                          Thread Group: main : max priority:10, demon:false

                          Thread: DestroyJavaVM : priority:5, demon:false

                          Thread Group: jboss : max priority:10, demon:false

                          Thread: Thread-0 : priority:5, demon:true
                          Thread: Listener:43931 : priority:5, demon:true
                          Thread: ScannerThread : priority:5, demon:true
                          Thread: Thread-3 : priority:5, demon:true
                          Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false
                          Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true
                          Thread: JBossMQ Cache Reference Softner : priority:5, demon:true
                          Thread: Thread-4 : priority:5, demon:true
                          Thread: HSQLDB Timer @12297d7 : priority:5, demon:true
                          Thread: Thread-6 : priority:5, demon:true
                          Thread: JCA PoolFiller : priority:5, demon:false
                          Thread: Thread-7 : priority:5, demon:true
                          Thread: API-Thread-8 : priority:5, demon:false
                          Thread: Thread-10 : priority:5, demon:false
                          Thread: SESSION-Thread-11 : priority:5, demon:false
                          Thread: Process-Scheduler : priority:5, demon:false
                          Thread: Process-Scheduler-1 : priority:5, demon:false
                          Thread: Process-Scheduler-2 : priority:5, demon:false
                          Thread: Thread-12 : priority:5, demon:false
                          Thread: PROXY-SESSION-Thread-13 : priority:5, demon:false
                          Thread: POLLER-THREAD : priority:5, demon:false
                          Thread: Thread-15 : priority:5, demon:false
                          Thread: Thread-16 : priority:5, demon:false
                          Thread: Thread-17 : priority:5, demon:true
                          Thread: Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:1/2d65a4cfa5d382c9af16bbfac51c7193 destination=QUEUE.CPSAsynchronousProvisioningQueue messageSelector=null Local Create] id=1 : priority:5, demon:false
                          Thread: Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:3/0a62301c4debbabf02308774a91376ba destination=TOPIC.CPSConfigurationMessageDispatcher messageSelector=null Local Create] id=2 : priority:5, demon:false
                          Thread: IdleRemover : priority:5, demon:true
                          Thread: API-Thread-18 : priority:5, demon:false
                          Thread: Thread-19 : priority:5, demon:false
                          Thread: JBossLifeThread : priority:5, demon:false
                          Thread: Thread-20 : priority:5, demon:true
                          Thread: Thread-21 : priority:5, demon:false
                          Thread: Listener:44140 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-1 : priority:5, demon:true
                          Thread: TP-Processor1 : priority:5, demon:true
                          Thread: TP-Processor2 : priority:5, demon:true
                          Thread: TP-Processor3 : priority:5, demon:true
                          Thread: TP-Processor4 : priority:5, demon:true
                          Thread: TP-Monitor : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-2 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-3 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-4 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-5 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-6 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-7 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-8 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-9 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-10 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-11 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-12 : priority:5, demon:true

                          Thread Group: JBoss Pooled Threads : max priority:10, demon:false

                          Thread: WorkManager(2)-1 : priority:5, demon:true

                          Thread Group: System Threads : max priority:10, demon:false

                          Thread: JBoss System Threads(1)-1 : priority:5, demon:true
                          Thread: JBoss System Threads(1)-2 : priority:5, demon:true


                          Thread Group: JBossMQ Server Threads : max priority:10, demon:false

                          Thread: UILServerILService Accept Thread : priority:5, demon:false
                          Thread: Message Pushers-1 : priority:5, demon:true


                          Thread Group: JBossMQ Client Threads : max priority:10, demon:false


                          Thread Group: ASF Session Pool Threads : max priority:10, demon:false

                          Thread: JMS SessionPool Worker-0 : priority:5, demon:true


                          Thread Group: RMI Runtime : max priority:10, demon:false

                          =============================================

                          Thread Dump 2 when the response is extremely delayed

                          =============================================
                          Total Threads: 72
                          Total Thread Groups: 9

                          Thread Group: system : max priority:10, demon:false

                          Thread: Reference Handler : priority:10, demon:true
                          Thread: Finalizer : priority:8, demon:true
                          Thread: Signal Dispatcher : priority:10, demon:true
                          Thread: AdapterThread : priority:10, demon:true
                          Thread: CompilerThread0 : priority:10, demon:true
                          Thread: CompilerThread1 : priority:10, demon:true
                          Thread: RMI TCP Accept-0 : priority:5, demon:true
                          Thread: RMI Reaper : priority:5, demon:false
                          Thread: GC Daemon : priority:2, demon:true
                          Thread: RMI TCP Accept-4444 : priority:5, demon:true
                          Thread: RMI TCP Accept-0 : priority:5, demon:true

                          Thread Group: main : max priority:10, demon:false

                          Thread: DestroyJavaVM : priority:5, demon:false

                          Thread Group: jboss : max priority:10, demon:false

                          Thread: Thread-0 : priority:5, demon:true
                          Thread: Listener:43931 : priority:5, demon:true
                          Thread: ScannerThread : priority:5, demon:true
                          Thread: Thread-3 : priority:5, demon:true
                          Thread: PooledInvokerAcceptor#0-4445 : priority:5, demon:false
                          Thread: ContainerBackgroundProcessor[StandardEngine[jboss.web]] : priority:5, demon:true
                          Thread: JBossMQ Cache Reference Softner : priority:5, demon:true
                          Thread: Thread-4 : priority:5, demon:true
                          Thread: HSQLDB Timer @12297d7 : priority:5, demon:true
                          Thread: Thread-6 : priority:5, demon:true
                          Thread: JCA PoolFiller : priority:5, demon:false
                          Thread: Thread-7 : priority:5, demon:true
                          Thread: API-Thread-8 : priority:5, demon:false
                          Thread: Thread-10 : priority:5, demon:false
                          Thread: SESSION-Thread-11 : priority:5, demon:false
                          Thread: Process-Scheduler : priority:5, demon:false
                          Thread: Process-Scheduler-1 : priority:5, demon:false
                          Thread: Process-Scheduler-2 : priority:5, demon:false
                          Thread: Thread-12 : priority:5, demon:false
                          Thread: PROXY-SESSION-Thread-13 : priority:5, demon:false
                          Thread: POLLER-THREAD : priority:5, demon:false
                          Thread: Thread-15 : priority:5, demon:false
                          Thread: Thread-16 : priority:5, demon:false
                          Thread: Thread-17 : priority:5, demon:true
                          Thread: Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:1/2d65a4cfa5d382c9af16bbfac51c7193 destination=QUEUE.CPSAsynchronousProvisioningQueue messageSelector=null Local Create] id=1 : priority:5, demon:false
                          Thread: Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:3/0a62301c4debbabf02308774a91376ba destination=TOPIC.CPSConfigurationMessageDispatcher messageSelector=null Local Create] id=2 : priority:5, demon:false
                          Thread: IdleRemover : priority:5, demon:true
                          Thread: API-Thread-18 : priority:5, demon:false
                          Thread: Thread-19 : priority:5, demon:false
                          Thread: JBossLifeThread : priority:5, demon:false
                          Thread: Thread-20 : priority:5, demon:true
                          Thread: Thread-21 : priority:5, demon:false
                          Thread: Listener:44140 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-1 : priority:5, demon:true
                          Thread: TP-Processor1 : priority:5, demon:true
                          Thread: TP-Processor2 : priority:5, demon:true
                          Thread: TP-Processor3 : priority:5, demon:true
                          Thread: TP-Processor4 : priority:5, demon:true
                          Thread: TP-Monitor : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-2 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-3 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-4 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-5 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-6 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-7 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-8 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-9 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-10 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-11 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-12 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-13 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-14 : priority:5, demon:true
                          Thread: http-0.0.0.0-8081-15 : priority:5, demon:true

                          Thread Group: JBoss Pooled Threads : max priority:10, demon:false

                          Thread: WorkManager(2)-1 : priority:5, demon:true

                          Thread Group: System Threads : max priority:10, demon:false

                          Thread: JBoss System Threads(1)-1 : priority:5, demon:true
                          Thread: JBoss System Threads(1)-2 : priority:5, demon:true


                          Thread Group: JBossMQ Server Threads : max priority:10, demon:false

                          Thread: UILServerILService Accept Thread : priority:5, demon:false
                          Thread: Message Pushers-1 : priority:5, demon:true


                          Thread Group: JBossMQ Client Threads : max priority:10, demon:false


                          Thread Group: ASF Session Pool Threads : max priority:10, demon:false

                          Thread: JMS SessionPool Worker-0 : priority:5, demon:true


                          Thread Group: RMI Runtime : max priority:10, demon:false
                          =============================================