idle time goes to 0 in an instant and stays that way

[nwc]

First 1 processor went to 0% idle and then 73 seconds later the 2nd processor also went to 0%, and stayed down.

Looking at our vmstat logs, this happened yesterday at 2:06 PM, but amazingly, nobody complained about the system being unusually slow until this afternoon, at which point we restarted JBoss. Here are the facts:

- Throughout this time, the system still responded to requests, most of the time very quickly, but sometimes not at all.

- vmstat shows an almost constant 0% idle, the only exceptions being about 10 blibs per minute going up to 7% idle, and going to 50% idle for 2.5 seconds at exactly-60-second intervals.

- We have a two processor machine.

- At the time we restarted, the JVM was running about 490 threads.

- We have maxProcessors set to 200.

- Along with the drop to 0% idle, the context switches reported by vmstat jumped from an average of about 1000 per second to an average of about 300,000 per second.

- This is the first time this has happened on a system that has been running under JBoss for 2.5 years, with steadily increasing traffic.

- Last application patch was a minor one two days ago; last major patch was 2 months ago.

- DB load and web server load during this whole time was normal.

What could it be? An infinite loop in newly deployed code? A JBoss configuration setting that's reached a threshhold? An OS setting?


Many thanks,

nwc

[slaboure]

- which JVM and which OS (and threading model)?

could that be a JVM/OS threading issue?

[nwc]

- which JVM and which OS (and threading model)?

Red Hat Enterprise Linux ES release 3 (Taroon)

java version "1.4.2_03"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)

JBoss 3.2.3

[nwc]

Oh, and:

Linux version 2.4.21-4.0.1.ELsmp ([e-mail]) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #1 SMP Thu Oct 23 01:21:42 EDT 2003

[genman]

I would turn on debug logging and get stack traces. If you get a feeling for what your process normally does during the time it changes behavior, a trace might pinpoint something.

Also, check using "lsof" and see if your process is running out of files.

[nwc]

"genman" wrote:

I would turn on debug logging and get stack traces. If you get a feeling for what your process normally does during the time it changes behavior, a trace might pinpoint something.

Also, check using "lsof" and see if your process is running out of files.

Thanks for the suggestions.

Our application does use (lots of) temporary files quite intensively, but if there was a file handle problem, shouldn't there be something about it recorded in a log file somewhere?