JBOSS dies every 24 hours?

[dwave]

Hi,

I'm running jboss3.0.4 on redhat7.2 and postgresql.

After about 24 hours of running jboss seems to die. That is, I have a simple daemon that runs every 30 seconds and samples the number of processes for apache, postgresql and jboss. About every 24 hours from starting jboss, the jboss process count suddenly goes to 0 (from about 71) postgresql and apache stay the same. This happens even when there is no load - just a server sitting there. My daemon restarts jboss and it comes back up again, but... it will die again after another 24 hours. This has gone on for 6 days straight.

I don't know jboss that well, and the logs don't show any real errors or exceptions near the critical moments. So, my question is this: is there some process (or setting) that regularly kills jboss? (much like the logrotate would SIGHUP apache)
Also, what sort of error in apache or postgresql could cause jboss to die suddenly? Where would I look to get evidence of such an error?

thanks in advance,

-dave

[juhalindfors]

There's nothing in JBoss that would shutdown the server automatically.

How does the process die? Is the server still responsive when the process is killed or does it seem like the app server has frozen and then you kill the process?

[dwave]

Thanks for the help.

Here's what happens - the result of:

ps -wax | grep java | grep -v grep | wc -l

is 0. At no more than 30 seconds earlier, it was 71. At this point, browsing any web page (e.g. a .jsp) will return 500 error. The database (postgresql) and apache seem to be running fine:

ps -wax | grep post | grep -v grep | wc -l
ps -wax | grep http | grep -v grep | wc -l

both are nonzero (12 and 9 I think). So, the machine is still running (it has not crashed), I just don't have any jboss...

The number of open files is pretty large in general:

lsof -c java | wc -l

starts out at 10651 - I haven't yet seen this number when jboss stops - in the next 24 hours I'll know. An fd leak seems unlikely, since the machine continues to run, but I'll try anything!

-dave

[mikefinn]

It's a major long shot, but make sure there's nothing in cron or anacron removing files from your temp dir, such as tmpwatch. This burned us a while back. We were getting 404s every 11 days (!), and it turned out tmpwatch was purging /tmp, the default place where JBoss puts temp files (System property java.io.tmpdir).

mike

[triathlon98]

What is your configuration? OS, JVM, processor?

Are there any extra files appearing in your jboss bin directory (like something starting with "hs_err_".

We had similar problems when running JBoss on a system with VIA C3 processor, and it appeared to be related that.

Joachim

[dwave]

YES! I have an hs_err_pidxxxx.log file for every time Jboss has died!
I'm using:
jboss-3.0.4
j2sdk 1.4.1_01-fcs (from rpm -q)
on redhat 2.4.7-10 (redhat linux 7.1 2.96-98)
on a IBM eServer 1U rack-server 1 CPU (Intel Pentium III 1GHz)

each hs_err file says this:

Unexpected Signal : 11 occurred at PC=0x40533620
Library=/usr/java/j2sdk1.4.1_01/jre/lib/i386/server/libjvm.so

Then a lot of other wonderful stuff, and then finally:

# HotSpot Virtual Machine Error : 11
# Error ID : 4F530E43505002E6
# Please report this error at
# http:// ...
# Java VM: Java HotSpot(TM) Server VM (1.4.1_01-b01 mixed mode)

How did you fix the problem?

Do I have a bad combination of jboss/jvm/os ?

The IBM hardware seems standard enough...

Thank you very much, indeed!

-dave

[triathlon98]

Bad luck, can't help in this case. Our only option was to switch systems to something with a different processor.

Joachim

[dwave]

Thanks though,

I'm going to try going back to 1.3.1 - don't think I really need 1.4.

-dave