8 Replies Latest reply on Sep 13, 2012 10:49 PM by Johan Vermeij

    jboss as 7.1.1 final stop or freezes when starting it under AIX

    samy omar Newbie

      I am trying to run Jboss as under our production server but it stopped while starting.

       

      OS: AIX 5.3

      JDK: 6 (64 bit)

      Jboss as: 7.1.1

       

      attached the server.log and boot.log files it may help

       

      steps I did :

       

      1- I copied the unzipped folder for jboss as to its directory under AIX (I unzipped it under windows with winrar instead of using TAR command, as it trim long file names).

      2- I tested the Java_home by typing on server consol, java -jar --> it show the correct installed java 6 (64 bit) path

      $ java -version

      java version "1.6.0"

      Java(TM) SE Runtime Environment (build pap6460sr10fp1-20120321_01(SR10 FP1))

      IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr10fp1-2012020

      2_101568 (JIT enabled, AOT enabled)

      J9VM - 20120202_101568

      JIT  - r9_20111107_21307ifx1

      GC   - 20120202_AA)

      JCL  - 20120320_01

       

      3- then I executed the standalone.sh file

       

      The consol printed this below, it stopped after printing "Bound data source [java:jboss/datasources/ExampleDS]"

      and get freezed.

       

       

       

      $ standalone.sh

      =========================================================================

        JBoss Bootstrap Environment

        JBOSS_HOME: /app/chub/jboss-as-7.1.1.Final

        JAVA: /usr/java6_64/bin/java

        JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx

      512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.w

      arning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInter

      val=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=tr

      ue -Djboss.server.default.config=standalone.xml

       

      ========================================================================

       

      17:03:32,701 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA

      17:03:33,966 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA

      17:03:34,315 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" sta

      rting

      17:03:39,089 INFO  [org.xnio] XNIO Version 3.0.3.GA

      17:03:39,152 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA

      17:03:39,193 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.3.GA

      17:03:39,239 INFO  [org.jboss.as.server] JBAS015888: Creating http management se

      rvice using socket-binding (management-http)

      17:03:39,508 INFO  [org.jboss.as.configadmin] JBAS016200: Activating ConfigAdmin

      Subsystem

      17:03:39,539 INFO  [org.jboss.as.clustering.infinispan] JBAS010280: Activating I

      nfinispan subsystem.

      17:03:39,563 INFO  [org.jboss.as.naming] JBAS011800: Activating Naming Subsystem

      17:03:39,661 INFO  [org.jboss.as.webservices] JBAS015537: Activating WebServices

      Extension

      17:03:39,799 INFO  [org.jboss.as.osgi] JBAS011940: Activating OSGi Subsystem

      17:03:39,823 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log han

      dlers

      17:03:40,168 INFO  [org.jboss.as.security] (MSC service thread 1-4) JBAS013100:

      Current PicketBox version=4.0.7.Final

      17:03:40,481 INFO  [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: St

      arting Naming Service

      17:03:41,848 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS01

      5400: Bound mail session [java:jboss/mail/Default]

      17:03:42,171 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC se

      rvice thread 1-2) JBoss Web Services - Stack CXF Server 4.0.2.GA

      17:03:42,780 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerServic

      e Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Dr

      iver (version 1.3)

      17:03:42,823 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread

      1-4) Starting Coyote HTTP/1.1 on http--127.0.0.1-8080

      17:03:44,062 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100:

      Listening on /127.0.0.1:9999

      17:03:44,138 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread

      1-3) JBAS015012: Started FileSystemDeploymentService for directory /app/chub/jbo

      ss-as-7.1.1.Final/standalone/deployments

      17:03:44,312 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service

      thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]

       

       

       

       

       

       

      >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

      Attached:

      - server.log

      - boot.log

      - Thread stack dump file (created using kill -QUIT <pid> on another terminal)

       

      As a result I cant shudown the app server propeply (as its not started correctly).

      what can I do with this strange and very annoying issue ???

      your help is highly ppretiated .............

       

      Regards

      Samy

        • 2. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
          samy omar Newbie

          I attached the thread stack dump to my original post,

          I opened another terminal  and used  this comman  to create a thread dump

          kill -QUIT <pid>

           

          the result:

          1- Something strage happened, on the terminal where I started the jboss, first it created the thread dump file (expected), But then the jboss server continued its loading and started up successfully.

          Here are the console output

           

          $ standalone.sh

          =========================================================================

             JBoss Bootstrap Environment

             JBOSS_HOME: /app/chub/jboss-as-7.1.1.Final

             JAVA: /usr/java6_64/bin/java

             JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx

          512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.w

          arning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInter

          val=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=tr

          ue -Djboss.server.default.config=standalone.xml

          =========================================================================

          17:22:45,778 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA

          17:22:50,270 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA

          17:22:50,625 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" sta

          rting

          17:23:03,018 INFO  [org.jboss.as.server] JBAS015888: Creating http management se

          rvice using socket-binding (management-http)

          17:23:03,087 INFO  [org.xnio] XNIO Version 3.0.3.GA

          17:23:03,319 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA

          17:23:03,415 INFO  [org.jboss.as.logging] JBAS011502: Removing bootstrap log han

          dlers

          17:23:03,429 INFO  [org.jboss.as.configadmin] (ServerService Thread Pool -- 26)

          JBAS016200: Activating ConfigAdmin Subsystem

          17:23:03,490 INFO  [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011

          940: Activating OSGi Subsystem

          17:23:03,499 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 44) JBA

          S013101: Activating Security Subsystem

          17:23:03,533 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 48)

          JBAS015537: Activating WebServices Extension

          17:23:03,544 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS0

          11800: Activating Naming Subsystem

          17:23:03,674 INFO  [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting

          version 3.2.3.GA

          17:23:03,774 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Po

          ol -- 31) JBAS010280: Activating Infinispan subsystem.

          17:23:04,464 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013100:

          Current PicketBox version=4.0.7.Final

          17:23:04,526 INFO  [org.jboss.as.connector] (MSC service thread 1-3) JBAS010408:

          Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)

          17:23:05,412 INFO  [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: St

          arting Naming Service

          17:23:06,969 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS01

          5400: Bound mail session [java:jboss/mail/Default]

          17:23:07,075 INFO  [org.jboss.as.connector.subsystems.datasources] (ServerServic

          e Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Dr

          iver (version 1.3)

          17:23:08,778 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread

          1-1) Starting Coyote HTTP/1.1 on http--0.0.0.0-8080

          17:23:10,755 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC se

          rvice thread 1-2) JBoss Web Services - Stack CXF Server 4.0.2.GA

          17:23:14,931 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service

          thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]

          17:23:19,064 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread

          1-3) JBAS015012: Started FileSystemDeploymentService for directory /app/chub/jbo

          ss-as-7.1.1.Final/standalone/deployments

          17:23:20,944 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100:

          Listening on /0.0.0.0:4447

          JVMDUMP039I Processing dump event "user", detail "" at 2012/09/04 17:45:38 - ple

          ase wait.

          JVMDUMP032I JVM requested Java dump using '/app/chub/jboss-as-7.1.1.Final/bin/ja

          vacore.20120904.174538.1224846.0001.txt' in response to an event

          JVMDUMP010I Java dump written to /app/chub/jboss-as-7.1.1.Final/bin/javacore.201

          20904.174538.1224846.0001.txt

          JVMDUMP013I Processed dump event "user", detail "".

          17:45:40,200 INFO  [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100:

          Listening on /127.0.0.1:9999

          17:45:40,291 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin con

          sole listening on http://0.0.0.0:9990

          17:45:40,294 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS

          7.1.1.Final "Brontes" started in 1381698ms - Started 133 of 209 services (75 ser

          vices are passive or on-demand)

           

           

          I feel that its a JVM versions issue, but I dont know where is the issue or how to workaround.

          Thanks for your help.....

          • 3. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
            Nicklas Karlsson Master

            I noticed you alredy found the thread with the similar problem on 6.x, it is most likely a JVM bug.

            • 4. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
              samy omar Newbie

              I found the solution and like to share the conclusion with community.

               

              Issue:

              Jboss freez or stop while starting it up, under AIX 5.3 OS

               

              root cause reason.

              incompitability between Jboss as and JDK6 installed under AIX OS, due to a bug in JDK

               

              solution/workaround

              - As you know JDK release for AIX contains many SRs, SR7 is the working version of JDK6.

              the current SR10 is not working and I tried SR1 also and not working.

              so if you have installed JDK6 SR10, please consider downgrading to JDK6 SR7 which worked with me perfectly.

               

              you can install JDK6 SR7 from this link

              http://www.ibm.com/developerworks/java/jdk/aix/j632/Java6.fixes.html

               

              Note:

              1- I did not try SR11 yet.

              2- this issue occured also for Jboss 6 (https://community.jboss.org/thread/200128)

              • 5. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
                Johan Vermeij Newbie

                I had the same issue where JBoss 7.1 was hanging on startup under AIX.

                I followed your advise (use SR7) and this fixed the startup problem, thank you for your post.

                Now I am stuck with all kinds of strange behaviour, see https://community.jboss.org/message/758989#758989

                I will try SR10 and SR11 and post the result here.

                • 6. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
                  Johan Vermeij Newbie

                  We tested with SR2, SR7, SR10 and SR 11

                  Under SR2, SR10 and SR11 JBoss 7.1.1 "hangs" at startup during remoting initialization.

                  Only SR7 was workable but very buggy.

                   

                  I attach the server log (DEBUG level) and standalone-full.xml

                   

                  The exact same configuration works perfectly under CentOS 6, Windows XP and Windows 7

                   

                  =========================================================================

                   

                    JBoss Bootstrap Environment

                   

                    JBOSS_HOME: /usr/share/jboss/jboss-as-7.1.1.Final

                   

                    JAVA: /usr/java6_64/bin/java

                   

                    JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms5120m -Xmx5120m -XX:MaxPermSize=1048m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

                   

                  =========================================================================

                   

                  15:42:38,055 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA

                  15:42:38,275 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA

                  15:42:38,333 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting

                  15:42:39,748 INFO  [org.xnio] XNIO Version 3.0.3.GA

                  15:42:39,751 INFO  [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)

                  15:42:39,761 INFO  [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA

                  15:42:39,773 INFO  [org.jboss.remoting] JBoss Remoting version 3.2.3.GA

                  15:42:40,298 WARN  [org.jboss.as.messaging] (MSC service thread 1-14) JBAS011600: AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal

                  HANGS HERE (detail in the attached server.log)

                  • 7. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
                    jaikiran pai Master

                    Has anyone in this thread reported this issue to the JDK vendor? If not, could one of you please do so? There's no point sticking to a lower bug fix version and it looks like migrating to a higher version isn't going to work either. So they should be made aware of this issue to get this fixed.

                    1 of 1 people found this helpful
                    • 8. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX
                      Johan Vermeij Newbie

                      I have double checked everything and only SR7 works but with unexpected results (weird behaviour).

                      Test abandoned.