Cannot fully start jboss server 4.2.0. on centos 5.2

[jurerai]

Hi,

We have a problem with starting Jboss server 4.2.0 on centos 5.2.

JBOSS server is a part of EMC documentum instalation and is called JMS but it is still basic JBOSS server.

The startup stalls at:
14:17:39,521 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:jboss-log4j.xml

Does anybody know the answer to this problem? Any direction in solving is appreciated.

Here is full output during startup of method server:


[dmadmin@barbara ~]$ ./startms
=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /dctm/shared/jboss4.2.0

JAVA: /dctm/shared/jboss4.2.0/jdk/bin/java

JAVA_OPTS: -Dprogram.name=run.sh -server -Xms256m -Xmx256m -XX:PermSize=64m -XX:MaxPermSize=256m -Xss256k -XX:+DisableExplicitGC -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.net.preferIPv4Stack=true

CLASSPATH: /dctm/shared/jboss4.2.0/bin/run.jar:/dctm/shared/jboss4.2.0/jdk/lib/tools.jar

=========================================================================

14:17:37,389 INFO [Server] Starting JBoss (MX MicroKernel)...
14:17:37,390 INFO [Server] Release ID: JBoss [EAP] 4.2.0.GA_CP01 (build: SVNTag=JBPAPP_4_2_0_GA_CP01 date=200709131706)
14:17:37,391 INFO [Server] Home Dir: /dctm/shared/jboss4.2.0
14:17:37,391 INFO [Server] Home URL: file:/dctm/shared/jboss4.2.0/
14:17:37,392 INFO [Server] Patch URL: null
14:17:37,392 INFO [Server] Server Name: DctmServer_MethodServer
14:17:37,392 INFO [Server] Server Home Dir: /dctm/shared/jboss4.2.0/server/DctmServer_MethodServer
14:17:37,392 INFO [Server] Server Home URL: file:/dctm/shared/jboss4.2.0/server/DctmServer_MethodServer/
14:17:37,392 INFO [Server] Server Log Dir: /dctm/shared/jboss4.2.0/server/DctmServer_MethodServer/log
14:17:37,392 INFO [Server] Server Temp Dir: /dctm/shared/jboss4.2.0/server/DctmServer_MethodServer/tmp
14:17:37,393 INFO [Server] Root Deployment Filename: jboss-service.xml
14:17:37,616 INFO [ServerInfo] Java version: 1.5.0_12,Sun Microsystems Inc.
14:17:37,616 INFO [ServerInfo] Java VM: Java HotSpot(TM) Server VM 1.5.0_12-b04,Sun Microsystems Inc.
14:17:37,616 INFO [ServerInfo] OS-System: Linux 2.6.18-92.1.18.el5,i386
14:17:37,883 INFO [Server] Core system initialized
14:17:39,519 INFO [WebService] Using RMI server codebase: http://barbara.marg.si:9083/
14:17:39,521 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:jboss-log4j.xml

[jaikiran]

Did you do any changes to the %JBOSS_HOME%/server/< serverName>/conf/jboss-lo4j.xml file? And does the %JBOSS_HOME%/server/< serverName>/log/server.log show any progress? And finally, does the http request work http://localhost:8080/

[jaikiran]

"jaikiran" wrote:
And finally, does the http request work http://localhost:8080/

Assuming, you haven't changed the http port. Else, you the correct http port and see if you can access the server.

14:17:39,519 INFO [WebService] Using RMI server codebase: http://barbara.marg.si:9083/

Try accessing http://barbara.marg.si:< httpPort>/

[jurerai]

On http://localhost:9080/ we can access:

JBoss Management

* Tomcat status (full) (XML)
* JMX Console
* JBoss Web Console

Did you do any changes to the %JBOSS_HOME%/server/< serverName>/conf/jboss-lo4j.xml file?

We didn't change anything, but this is a part of changed instalation files of EMC Documentum server and I presume that they have changed something. I would like to know in which direction we can look in to to troubleshoot the problem.

And does the %JBOSS_HOME%/server/< serverName>/log/server.log show any progress?

I have posted log file on this address:

http://www.4shared.com/file/76195567/7880f7e/server.html

I can see errors, but I don't know what to think about them. The main problem is that Jboss is in this configuration just as method server running methods, the application server is on different server on tomcat and is starting perfectly. But when the application server is trying to run methods on JBoss there are errors as:

---
An error has occurred.

invokeMethod() failed while calling: onLogin
null
class java.lang.NullPointerException

After closing this window, press the Refresh or Reload button on your browser to continue.
---

We think that this is a problem with JBoss not starting correctly.

Thank you and best regards, Jure

[jurerai]

Update:

When I set debug = true in jboss-log4j.xml configuration file I got this return in line: Adding appender named ... Hope this would help.

log4j: Adding appender named [ASYNC] to category [root].
Exception in thread "Session pool worker" java.lang.NullPointerException
at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:623)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:467)
at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:408)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at com.documentum.fc.client.impl.session.Session.disconnect(Session.java:237)
at com.documentum.fc.client.impl.session.SessionPool.disconnectSessions(SessionPool.java:111)
at com.documentum.fc.client.impl.session.SessionPool.flush(SessionPool.java:134)
at com.documentum.fc.client.impl.session.SessionPool.flushAllPools(SessionPool.java:190)
at com.documentum.fc.client.impl.session.SessionPool$ExpirationThread.run(SessionPool.java:246)

[jaikiran]

Please post the contents of the jboss-log4j.xml file.

While posting xml content or code or logs, please remember to wrap it in a code block by using the Code button in the message editor window. Please use the Preview button before posting, to ensure your post is correctly formatted.

[jurerai]

Here it is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<!-- ===================================================================== -->
<!-- -->
<!-- Log4j Configuration -->
<!-- -->
<!-- ===================================================================== -->

<!-- $Id: jboss-log4j.xml,v 1.0, 2008-01-07 23:40:45Z, Jason Huang$ -->

<!--
 | For more configuration infromation and examples see the Jakarta Log4j
 | owebsite: http://jakarta.apache.org/log4j
 -->

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="true">

 <!-- ================================= -->
 <!-- Preserve messages in a local file -->
 <!-- ================================= -->

 <!-- A time/date based rolling appender -->
 <appender name="FILE" class="org.jboss.logging.appender.DailyRollingFileAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="File" value="${jboss.server.log.dir}/server.log"/>
 <param name="Append" value="false"/>

 <!-- Rollover at midnight each day -->
 <param name="DatePattern" value="'.'yyyy-MM-dd"/>

 <!-- Rollover at the top of each hour
 <param name="DatePattern" value="'.'yyyy-MM-dd-HH"/>
 -->

 <layout class="org.apache.log4j.PatternLayout">
 <!-- The default pattern: Date Priority [Category] Message\n -->
 <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>

 <!-- The full pattern: Date MS Priority [Category] (Thread:NDC) Message\n
 <param name="ConversionPattern" value="%d %-5r %-5p [%c] (%t:%x) %m%n"/>
 -->
 </layout>
 </appender>

 <!-- A size based file rolling appender
 <appender name="FILE" class="org.jboss.logging.appender.RollingFileAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="File" value="${jboss.server.log.dir}/server.log"/>
 <param name="Append" value="false"/>
 <param name="MaxFileSize" value="500KB"/>
 <param name="MaxBackupIndex" value="1"/>

 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
 </layout>
 </appender>
 -->

 <!-- ============================== -->
 <!-- Append messages to the console -->
 <!-- ============================== -->
 <!--
 <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="Target" value="System.out"/>
 <param name="Threshold" value="INFO"/>

 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}] %m%n"/>
 </layout>
 </appender>
 -->

 <!-- ====================== -->
 <!-- More Appender examples -->
 <!-- ====================== -->

 <!-- Buffer events and log them asynchronously -->
 <appender name="ASYNC" class="org.apache.log4j.AsyncAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <appender-ref ref="FILE"/>
 <!--
 <appender-ref ref="CONSOLE"/>
 <appender-ref ref="SMTP"/>
 -->
 </appender>

 <!-- EMail events to an administrator
 <appender name="SMTP" class="org.apache.log4j.net.SMTPAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="Threshold" value="ERROR"/>
 <param name="To" value="[e-mail]"/>
 <param name="From" value="[e-mail]"/>
 <param name="Subject" value="JBoss Sever Errors"/>
 <param name="SMTPHost" value="localhost"/>
 <param name="BufferSize" value="10"/>
 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="[%d{ABSOLUTE},%c{1}] %m%n"/>
 </layout>
 </appender>
 -->

 <!-- Syslog events
 <appender name="SYSLOG" class="org.apache.log4j.net.SyslogAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="Facility" value="LOCAL7"/>
 <param name="FacilityPrinting" value="true"/>
 <param name="SyslogHost" value="localhost"/>
 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="[%d{ABSOLUTE},%c{1}] %m%n"/>
 </layout>
 </appender>
 -->

 <!-- Log events to JMS (requires a topic to be created)
 <appender name="JMS" class="org.apache.log4j.net.JMSAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="Threshold" value="ERROR"/>
 <param name="TopicConnectionFactoryBindingName" value="java:/ConnectionFactory"/>
 <param name="TopicBindingName" value="topic/MyErrorsTopic"/>
 </appender>
 -->

 <!-- Log events through SNMP
 <appender name="TRAP_LOG" class="org.apache.log4j.ext.SNMPTrapAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="ImplementationClassName" value="org.apache.log4j.ext.JoeSNMPTrapSender"/>
 <param name="ManagementHost" value="127.0.0.1"/>
 <param name="ManagementHostTrapListenPort" value="162"/>
 <param name="EnterpriseOID" value="1.3.6.1.4.1.24.0"/>
 <param name="LocalIPAddress" value="127.0.0.1"/>
 <param name="LocalTrapSendPort" value="161"/>
 <param name="GenericTrapType" value="6"/>
 <param name="SpecificTrapType" value="12345678"/>
 <param name="CommunityString" value="public"/>
 <param name="ForwardStackTraceWithTrap" value="true"/>
 <param name="Threshold" value="DEBUG"/>
 <param name="ApplicationTrapOID" value="1.3.6.1.4.1.24.12.10.22.64"/>
 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="%d,%p,[%t],[%c],%m%n"/>
 </layout>
 </appender>
 -->

 <!-- Emit events as JMX notifications
 <appender name="JMX" class="org.jboss.monitor.services.JMXNotificationAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>

 <param name="Threshold" value="WARN"/>
 <param name="ObjectName" value="jboss.system:service=Logging,type=JMXNotificationAppender"/>

 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="%d %-5p [%c] %m"/>
 </layout>
 </appender>
 -->

 <!-- Clustering logging -->
 <!-- Uncomment the following to redirect the org.jgroups and
 org.jboss.ha categories to a cluster.log file. -->
 <!--
 <appender name="CLUSTER" class="org.jboss.logging.appender.RollingFileAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="File" value="${jboss.server.log.dir}/cluster.log"/>
 <param name="Append" value="false"/>
 <param name="MaxFileSize" value="500KB"/>
 <param name="MaxBackupIndex" value="1"/>

 <layout class="org.apache.log4j.PatternLayout">
 <param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
 </layout>
 </appender>
 -->

 <!-- ================ -->
 <!-- Limit categories -->
 <!-- ================ -->

 <category name="org.jboss.web">
 <priority value="INFO" />
 </category>

 <category name="org.jboss.ha">
 <priority value="INFO" />
 </category>

 <!-- Limit the org.apache category to INFO as its DEBUG is verbose -->
 <category name="org.apache">
 <priority value="WARN"/>
 </category>

 <!-- Limit the org.jboss.serial (jboss-serialization) to INFO as its DEBUG is verbose -->
 <category name="org.jboss.serial">
 <priority value="WARN"/>
 </category>

 <!-- Limit the org.jgroups category to WARN as its INFO is verbose -->
 <category name="org.jgroups">
 <priority value="WARN"/>
 </category>

 <!-- Limit the jacorb category to WARN as its INFO is verbose -->
 <category name="jacorb">
 <priority value="WARN"/>
 </category>

 <!-- Limit JBoss categories -->
 <category name="org.jboss">
 <priority value="INFO"/>
 </category>

 <!-- Limit the JSR77 categories -->
 <category name="org.jboss.management">
 <priority value="WARN"/>
 </category>

 <!-- Limit seam logging -->
 <category name="org.jboss.seam">
 <priority value="WARN"/>
 </category>

 <!-- Enable JBossWS message tracing
 <category name="org.jboss.ws.core.MessageTrace">
 <priority value="TRACE"/>
 </category>
 -->

 <!-- Decrease the priority threshold for the org.jboss.varia category
 <category name="org.jboss.varia">
 <priority value="DEBUG"/>
 </category>
 -->

 <!-- Show the evolution of the DataSource pool in the logs [inUse/Available/Max]
 <category name="org.jboss.resource.connectionmanager.JBossManagedConnectionPool">
 <priority value="TRACE"/>
 </category>
 -->

 <!--
 | An example of enabling the custom TRACE level priority that is used
 | by the JBoss internals to diagnose low level details. This example
 | turns on TRACE level msgs for the org.jboss.ejb.plugins package and its
 | subpackages. This will produce A LOT of logging output.
 |
 | Note: since jboss AS 4.2.x, the trace level is supported natively by
 | log4j, so although the custom org.jboss.logging.XLevel priority will
 | still work, there is no need to use it. The two examples that follow
 | will both enable trace logging.
 <category name="org.jboss.system">
 <priority value="TRACE" class="org.jboss.logging.XLevel"/>
 </category>
 <category name="org.jboss.ejb.plugins">
 <priority value="TRACE"/>
 </category>
 -->

 <!--
 | Logs these events to SNMP:
 - server starts/stops
 - cluster evolution (node death/startup)
 - When an EJB archive is deployed (and associated verified messages)
 - When an EAR archive is deployed

 <category name="org.jboss.system.server.Server">
 <priority value="INFO" />
 <appender-ref ref="TRAP_LOG"/>
 </category>

 <category name="org.jboss.ha.framework.interfaces.HAPartition.lifecycle">
 <priority value="INFO" />
 <appender-ref ref="TRAP_LOG"/>
 </category>

 <category name="org.jboss.deployment.MainDeployer">
 <priority value="ERROR" />
 <appender-ref ref="TRAP_LOG"/>
 </category>

 <category name="org.jboss.ejb.EJBDeployer">
 <priority value="INFO" />
 <appender-ref ref="TRAP_LOG"/>
 </category>

 <category name="org.jboss.deployment.EARDeployer">
 <priority value="INFO" />
 <appender-ref ref="TRAP_LOG"/>
 </category>

 -->

 <!-- ======================= -->
 <!-- Setup the Root category -->
 <!-- ======================= -->

 <root>
 <!-- <appender-ref ref="CONSOLE"/> -->
 <appender-ref ref="ASYNC"/>
 </root>

</log4j:configuration>

[jaikiran]

<!-- Buffer events and log them asynchronously -->
 <appender name="ASYNC" class="org.apache.log4j.AsyncAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <appender-ref ref="FILE"/>
 <!--
 <appender-ref ref="CONSOLE"/>
 <appender-ref ref="SMTP"/>
 -->
 </appender>

...

 <root>
 <!-- <appender-ref ref="CONSOLE"/> -->
 <appender-ref ref="ASYNC"/>
 </root>

This configuration shows that the default jboss-log4j.xml that was shipped has been changed (i guess intentionally).
* One of the change is to disable the CONSOLE appender for the root logger. That's the reason why you don't see anything logged on the console after the initial few logs
* The other change has configured the root logger to use the asynchronous appender to log to the server.log file.

[jurerai]

Could this be the reason that jboss could not start or should we look elsewhere?

Thank you,

Jure

[jaikiran]

"jurerai" wrote:
Could this be the reason that jboss could not start or should we look elsewhere?

Thank you,

Jure

I don't think that should stop the server from starting. Moreover, you mentioned that the server is accessible through the http request. So the server is up and your application probably is running into errors:

Exception in thread "Session pool worker" java.lang.NullPointerException
at org.jboss.mx.loading.RepositoryClassLoader.findClass(RepositoryClassLoader.java:623)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:467)
at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:408)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at com.documentum.fc.client.impl.session.Session.disconnect(Session.java:237)
at com.documentum.fc.client.impl.session.SessionPool.disconnectSessions(SessionPool.java:111)
at com.documentum.fc.client.impl.session.SessionPool.flush(SessionPool.java:134)
at com.documentum.fc.client.impl.session.SessionPool.flushAllPools(SessionPool.java:190)
at com.documentum.fc.client.impl.session.SessionPool$ExpirationThread.run(SessionPool.java:246)

The com.documentum.fc.client.impl.session.Session.disconnect is trying to load some class which is resulting in the NullPointerException. You will have to debug that piece of code.

[jurerai]

Jaikiran thank you for your help.