-
1. Re: JBOSS on CentOS - Logging Issue
onizukanne Jul 31, 2006 3:59 AM (in response to onizukanne)Two days and no replies yet... Unbelievable!
Has no one ever seen this issue before? Are we the first to encounter this? -
2. Re: JBOSS on CentOS - Logging Issue
jaikiran Jul 31, 2006 4:07 AM (in response to onizukanne)System.out.println is invoked as part of some code. Something like:
System.out.println("?");
You will have to search for the System.out.println statements in your code to find the culprit. -
3. Re: JBOSS on CentOS - Logging Issue
onizukanne Jul 31, 2006 4:32 AM (in response to onizukanne)Thanks jaikiran
That was the first search we ran, looked for EVERY single System.out.println message in all the code. There was no such record.
Besides like I mentioned, once the problem starts, even after we undeploy ALL our ears from the deploy folder, the logging continues. If it was due to a println message in our code, it would stop the moment the affected ear is undeployed.
So so strange -
4. Re: JBOSS on CentOS - Logging Issue
jaikiran Jul 31, 2006 4:41 AM (in response to onizukanne)Though i would not call this a ideal solution, you can change the threshold level of the CONSOLE appender to some value higher than INFO to supress these messages(and prevent the log files growing huge), till you get hold of the actual class from which this message is being logged.
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender"> <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"/> <param name="Target" value="System.out"/> <param name="Threshold" value="WARN"/> <layout class="org.apache.log4j.PatternLayout"> <!-- The default pattern: Date Priority [Category] Message\n --> <param name="ConversionPattern" value="%d{ISO8601} %-5p [%c] %m%n"/> </layout> </appender>
-
5. Re: JBOSS on CentOS - Logging Issue
onizukanne Jul 31, 2006 5:50 AM (in response to onizukanne)I'm afraid we already raised the threshold. Whatever the process though still continues running in the background, still causing the appserver to ultimately go out of memory, even without the appender logging anything.
Because of this, we spent the last 36 hours migrating the system from CentOS to Fedora Core (since all our other implementations are on Fedora and have been quite stable). All of a sudden the problem has started again -
6. Re: JBOSS on CentOS - Logging Issue
jaikiran Jul 31, 2006 6:00 AM (in response to onizukanne)Whatever the process though still continues running in the background, still causing the appserver to ultimately go out of memory, even without the appender logging anything.
This looks more than just a logging issue. In such cases, the best way to know what's going on is to obtain a thread dump. When you start seeing this behaviour, obtain the thread dump as mentioned here:
http://www.jboss.org/wiki/Wiki.jsp?page=StackTrace
The thread dump will display the list of threads and the activities that are going on(which might be causing the logging and the OutOfMemory issues). -
7. Re: JBOSS on CentOS - Logging Issue
onizukanne Jul 31, 2006 9:43 AM (in response to onizukanne)Thanks j
Would revert once I've reviewed the stack trace as advised -
8. Re: JBOSS on CentOS - Logging Issue
onizukanne Jul 31, 2006 2:36 PM (in response to onizukanne)Very many thanks for the insight j,
A thorough review of the thread dump threw up the culprit and we have updated the system accordingly.
Would watch it through tomorrow to monitor performance, but for the most part I believe the worst is over.
Thanks again mate