JBOSS on CentOS - Logging Issue

[onizukanne]

We've been running an application on liferay 3.6.1, running in JBoss4 on a Windows server.

On migrating the solution to a CentOS server, it appears to run alright at first. However, after about 30mins, we notice serious logging activity resulting in the log files growing to up to 4GB daily. We change the logger from the DailyRollingAppender to the RollingAppender to cut of at 2MB and the server generates about 8 2MB logfiles every second.

This is very strange, since once the problem starts, even after we undeploy all our ears and the liferay ear from the deploy folder, the logging continues. An extract of the logfile is like so:

2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,884 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?
2006-07-29 02:15:52,885 INFO [STDOUT] ?

Notice how many log lines are written every millisecond?

Any one with a similar experience? Any idea what could be responsible? Your ideas are highly welcome, this is really driving us crazy. This is happening on a life system and consumes lots and lots of system resources, causing us to need to restart the service every hour or so. URGENTLY need a solution.

Many thanks

[onizukanne]

Two days and no replies yet... Unbelievable!

Has no one ever seen this issue before? Are we the first to encounter this?

[jaikiran]

System.out.println is invoked as part of some code. Something like:

System.out.println("?");

You will have to search for the System.out.println statements in your code to find the culprit.

[onizukanne]

Thanks jaikiran

That was the first search we ran, looked for EVERY single System.out.println message in all the code. There was no such record.

Besides like I mentioned, once the problem starts, even after we undeploy ALL our ears from the deploy folder, the logging continues. If it was due to a println message in our code, it would stop the moment the affected ear is undeployed.

So so strange

[jaikiran]

Though i would not call this a ideal solution, you can change the threshold level of the CONSOLE appender to some value higher than INFO to supress these messages(and prevent the log files growing huge), till you get hold of the actual class from which this message is being logged.

<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
 <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/>
 <param name="Target" value="System.out"/>
 <param name="Threshold" value="WARN"/>
 <layout class="org.apache.log4j.PatternLayout">
 <!-- The default pattern: Date Priority [Category] Message\n -->
 <param name="ConversionPattern" value="%d{ISO8601} %-5p [%c] %m%n"/>
 </layout>
 </appender>

[onizukanne]

I'm afraid we already raised the threshold. Whatever the process though still continues running in the background, still causing the appserver to ultimately go out of memory, even without the appender logging anything.

Because of this, we spent the last 36 hours migrating the system from CentOS to Fedora Core (since all our other implementations are on Fedora and have been quite stable). All of a sudden the problem has started again

[jaikiran]

Whatever the process though still continues running in the background, still causing the appserver to ultimately go out of memory, even without the appender logging anything.

This looks more than just a logging issue. In such cases, the best way to know what's going on is to obtain a thread dump. When you start seeing this behaviour, obtain the thread dump as mentioned here:

http://www.jboss.org/wiki/Wiki.jsp?page=StackTrace

The thread dump will display the list of threads and the activities that are going on(which might be causing the logging and the OutOfMemory issues).

[onizukanne]

Thanks j

Would revert once I've reviewed the stack trace as advised

[onizukanne]

Very many thanks for the insight j,

A thorough review of the thread dump threw up the culprit and we have updated the system accordingly.

Would watch it through tomorrow to monitor performance, but for the most part I believe the worst is over.

Thanks again mate