8 Replies Latest reply on Sep 13, 2012 10:49 PM by johanvermeij

jboss as 7.1.1 final stop or freezes when starting it under AIX

samyomar82 Sep 4, 2012 2:10 PM

I am trying to run Jboss as under our production server but it stopped while starting.

OS: AIX 5.3

JDK: 6 (64 bit)

Jboss as: 7.1.1

attached the server.log and boot.log files it may help

steps I did :

1- I copied the unzipped folder for jboss as to its directory under AIX (I unzipped it under windows with winrar instead of using TAR command, as it trim long file names).

2- I tested the Java_home by typing on server consol, java -jar --> it show the correct installed java 6 (64 bit) path

$ java -version

java version "1.6.0"

Java(TM) SE Runtime Environment (build pap6460sr10fp1-20120321_01(SR10 FP1))

IBM J9 VM (build 2.4, JRE 1.6.0 IBM J9 2.4 AIX ppc64-64 jvmap6460sr10fp1-2012020

2_101568 (JIT enabled, AOT enabled)

J9VM - 20120202_101568

JIT - r9_20111107_21307ifx1

GC - 20120202_AA)

JCL - 20120320_01

3- then I executed the standalone.sh file

The consol printed this below, it stopped after printing "Bound data source [java:jboss/datasources/ExampleDS]"

and get freezed.

$ standalone.sh

=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /app/chub/jboss-as-7.1.1.Final

JAVA: /usr/java6_64/bin/java

JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx

512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.w

arning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInter

val=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=tr

ue -Djboss.server.default.config=standalone.xml

========================================================================

17:03:32,701 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA

17:03:33,966 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA

17:03:34,315 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" sta

rting

17:03:39,089 INFO [org.xnio] XNIO Version 3.0.3.GA

17:03:39,152 INFO [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA

17:03:39,193 INFO [org.jboss.remoting] JBoss Remoting version 3.2.3.GA

17:03:39,239 INFO [org.jboss.as.server] JBAS015888: Creating http management se

rvice using socket-binding (management-http)

17:03:39,508 INFO [org.jboss.as.configadmin] JBAS016200: Activating ConfigAdmin

Subsystem

17:03:39,539 INFO [org.jboss.as.clustering.infinispan] JBAS010280: Activating I

nfinispan subsystem.

17:03:39,563 INFO [org.jboss.as.naming] JBAS011800: Activating Naming Subsystem

17:03:39,661 INFO [org.jboss.as.webservices] JBAS015537: Activating WebServices

Extension

17:03:39,799 INFO [org.jboss.as.osgi] JBAS011940: Activating OSGi Subsystem

17:03:39,823 INFO [org.jboss.as.logging] JBAS011502: Removing bootstrap log han

dlers

17:03:40,168 INFO [org.jboss.as.security] (MSC service thread 1-4) JBAS013100:

Current PicketBox version=4.0.7.Final

17:03:40,481 INFO [org.jboss.as.naming] (MSC service thread 1-1) JBAS011802: St

arting Naming Service

17:03:41,848 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS01

5400: Bound mail session [java:jboss/mail/Default]

17:03:42,171 INFO [org.jboss.ws.common.management.AbstractServerConfig] (MSC se

rvice thread 1-2) JBoss Web Services - Stack CXF Server 4.0.2.GA

17:03:42,780 INFO [org.jboss.as.connector.subsystems.datasources] (ServerServic

e Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Dr

iver (version 1.3)

17:03:42,823 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service thread

1-4) Starting Coyote HTTP/1.1 on http--127.0.0.1-8080

17:03:44,062 INFO [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100:

Listening on /127.0.0.1:9999

17:03:44,138 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread

1-3) JBAS015012: Started FileSystemDeploymentService for directory /app/chub/jbo

ss-as-7.1.1.Final/standalone/deployments

17:03:44,312 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service

thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

Attached:

- server.log

- boot.log

- Thread stack dump file (created using kill -QUIT <pid> on another terminal)

As a result I cant shudown the app server propeply (as its not started correctly).

what can I do with this strange and very annoying issue ???

your help is highly ppretiated .............

Regards

Samy

server.log.zip 663 bytes
boot.log.zip 2.6 KB
javacore.20120904.174538.1224846.0001.txt.zip 127.1 KB

1. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

nickarls Sep 3, 2012 5:33 PM (in response to samyomar82)

Can you do a stack dump showing where it hangs?
Actions
2. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

samyomar82 Sep 4, 2012 2:13 PM (in response to nickarls)

I attached the thread stack dump to my original post,
I opened another terminal and used this comman to create a thread dump
kill -QUIT <pid>

the result:
1- Something strage happened, on the terminal where I started the jboss, first it created the thread dump file (expected), But then the jboss server continued its loading and started up successfully.
Here are the console output

$ standalone.sh
=========================================================================
   JBoss Bootstrap Environment
   JBOSS_HOME: /app/chub/jboss-as-7.1.1.Final
   JAVA: /usr/java6_64/bin/java
   JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms64m -Xmx
512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.w
arning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInter
val=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=tr
ue -Djboss.server.default.config=standalone.xml
=========================================================================
17:22:45,778 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA
17:22:50,270 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA
17:22:50,625 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" sta
rting
17:23:03,018 INFO [org.jboss.as.server] JBAS015888: Creating http management se
rvice using socket-binding (management-http)
17:23:03,087 INFO [org.xnio] XNIO Version 3.0.3.GA
17:23:03,319 INFO [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
17:23:03,415 INFO [org.jboss.as.logging] JBAS011502: Removing bootstrap log han
dlers
17:23:03,429 INFO [org.jboss.as.configadmin] (ServerService Thread Pool -- 26)
JBAS016200: Activating ConfigAdmin Subsystem
17:23:03,490 INFO [org.jboss.as.osgi] (ServerService Thread Pool -- 39) JBAS011
940: Activating OSGi Subsystem
17:23:03,499 INFO [org.jboss.as.security] (ServerService Thread Pool -- 44) JBA
S013101: Activating Security Subsystem
17:23:03,533 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 48)
JBAS015537: Activating WebServices Extension
17:23:03,544 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 38) JBAS0
11800: Activating Naming Subsystem
17:23:03,674 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting
version 3.2.3.GA
17:23:03,774 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Po
ol -- 31) JBAS010280: Activating Infinispan subsystem.
17:23:04,464 INFO [org.jboss.as.security] (MSC service thread 1-2) JBAS013100:
Current PicketBox version=4.0.7.Final
17:23:04,526 INFO [org.jboss.as.connector] (MSC service thread 1-3) JBAS010408:
Starting JCA Subsystem (JBoss IronJacamar 1.0.9.Final)
17:23:05,412 INFO [org.jboss.as.naming] (MSC service thread 1-2) JBAS011802: St
arting Naming Service
17:23:06,969 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) JBAS01
5400: Bound mail session [java:jboss/mail/Default]
17:23:07,075 INFO [org.jboss.as.connector.subsystems.datasources] (ServerServic
e Thread Pool -- 27) JBAS010403: Deploying JDBC-compliant driver class org.h2.Dr
iver (version 1.3)
17:23:08,778 INFO [org.apache.coyote.http11.Http11Protocol] (MSC service thread
1-1) Starting Coyote HTTP/1.1 on http--0.0.0.0-8080
17:23:10,755 INFO [org.jboss.ws.common.management.AbstractServerConfig] (MSC se
rvice thread 1-2) JBoss Web Services - Stack CXF Server 4.0.2.GA
17:23:14,931 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service
thread 1-3) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
17:23:19,064 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread
1-3) JBAS015012: Started FileSystemDeploymentService for directory /app/chub/jbo
ss-as-7.1.1.Final/standalone/deployments
17:23:20,944 INFO [org.jboss.as.remoting] (MSC service thread 1-2) JBAS017100:
Listening on /0.0.0.0:4447
JVMDUMP039I Processing dump event "user", detail "" at 2012/09/04 17:45:38 - ple
ase wait.
JVMDUMP032I JVM requested Java dump using '/app/chub/jboss-as-7.1.1.Final/bin/ja
vacore.20120904.174538.1224846.0001.txt' in response to an event
JVMDUMP010I Java dump written to /app/chub/jboss-as-7.1.1.Final/bin/javacore.201
20904.174538.1224846.0001.txt
JVMDUMP013I Processed dump event "user", detail "".
17:45:40,200 INFO [org.jboss.as.remoting] (MSC service thread 1-1) JBAS017100:
Listening on /127.0.0.1:9999
17:45:40,291 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin con
sole listening on http://0.0.0.0:9990
17:45:40,294 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS
7.1.1.Final "Brontes" started in 1381698ms - Started 133 of 209 services (75 ser
vices are passive or on-demand)

I feel that its a JVM versions issue, but I dont know where is the issue or how to workaround.
Thanks for your help.....
Actions
3. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

nickarls Sep 5, 2012 1:42 AM (in response to samyomar82)

I noticed you alredy found the thread with the similar problem on 6.x, it is most likely a JVM bug.
Actions
4. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

samyomar82 Sep 9, 2012 8:08 AM (in response to nickarls)

I found the solution and like to share the conclusion with community.

Issue:
Jboss freez or stop while starting it up, under AIX 5.3 OS

root cause reason.
incompitability between Jboss as and JDK6 installed under AIX OS, due to a bug in JDK

solution/workaround
- As you know JDK release for AIX contains many SRs, SR7 is the working version of JDK6.
the current SR10 is not working and I tried SR1 also and not working.
so if you have installed JDK6 SR10, please consider downgrading to JDK6 SR7 which worked with me perfectly.

you can install JDK6 SR7 from this link
http://www.ibm.com/developerworks/java/jdk/aix/j632/Java6.fixes.html

Note:
1- I did not try SR11 yet.
2- this issue occured also for Jboss 6 (https://community.jboss.org/thread/200128)
Actions
5. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

johanvermeij Sep 11, 2012 8:11 PM (in response to samyomar82)

I had the same issue where JBoss 7.1 was hanging on startup under AIX.
I followed your advise (use SR7) and this fixed the startup problem, thank you for your post.
Now I am stuck with all kinds of strange behaviour, see https://community.jboss.org/message/758989#758989
I will try SR10 and SR11 and post the result here.
Actions
6. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

johanvermeij Sep 12, 2012 4:53 AM (in response to johanvermeij)
We tested with SR2, SR7, SR10 and SR 11
Under SR2, SR10 and SR11 JBoss 7.1.1 "hangs" at startup during remoting initialization.
Only SR7 was workable but very buggy.

I attach the server log (DEBUG level) and standalone-full.xml

The exact same configuration works perfectly under CentOS 6, Windows XP and Windows 7

=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /usr/share/jboss/jboss-as-7.1.1.Final

JAVA: /usr/java6_64/bin/java

JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms5120m -Xmx5120m -XX:MaxPermSize=1048m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml

=========================================================================

15:42:38,055 INFO [org.jboss.modules] JBoss Modules version 1.1.1.GA
15:42:38,275 INFO [org.jboss.msc] JBoss MSC version 1.0.2.GA
15:42:38,333 INFO [org.jboss.as] JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
15:42:39,748 INFO [org.xnio] XNIO Version 3.0.3.GA
15:42:39,751 INFO [org.jboss.as.server] JBAS015888: Creating http management service using socket-binding (management-http)
15:42:39,761 INFO [org.xnio.nio] XNIO NIO Implementation Version 3.0.3.GA
15:42:39,773 INFO [org.jboss.remoting] JBoss Remoting version 3.2.3.GA
15:42:40,298 WARN [org.jboss.as.messaging] (MSC service thread 1-14) JBAS011600: AIO wasn't located on this platform, it will fall back to using pure Java NIO. If your platform is Linux, install LibAIO to enable the AIO journal
HANGS HERE (detail in the attached server.log)

standalone-full.xml 24.3 KB

server.log.zip 13.0 KB
Actions
7. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

jaikiran Sep 12, 2012 5:14 AM (in response to johanvermeij)

Has anyone in this thread reported this issue to the JDK vendor? If not, could one of you please do so? There's no point sticking to a lower bug fix version and it looks like migrating to a higher version isn't going to work either. So they should be made aware of this issue to get this fixed.
1 of 1 people found this helpful
Actions
8. Re: jboss as 7.1.1 final stop or freezes when starting it under AIX

johanvermeij Sep 13, 2012 10:49 PM (in response to jaikiran)

I have double checked everything and only SR7 works but with unexpected results (weird behaviour).
Test abandoned.
Actions

Go to original post