Application using all available memory

[cp]

We have recently launched a web site using

JBoss-3.2.1_Tomcat-4.1.24
Struts. Most recent stable build on July 10th.
MySQL 4.0
EJB 1.1
JDBC with mysql 4.0 support
Sun j2sdk1.4.2
Linux 7.3
PIII 1.8 GHz
1G RAM

We are experiencing intermittent slowdowns on our site that do not appear to be related to a particular page loading. I have noticed using "top" that the slowdown is related to new java processes being spawned. After a new process is spawned, the site becomes responsive again for several seconds (I can browse several pages) before slowing down again. Then it picks up when another process is spawned.

The java processes are very large and when the site is busy, there are usually four java processes using all available memory. Each of the processes is about 200M in size, even when there are less than four running.

I am a Sys Admin trying to assist the developers in determining the cause of the slowdowns. I wonder if anyone else has seen anything like this? Is this more likely a configuration problem or a problem with the code? If it is likely a code problem, what tool would you recommend for tracking it down?

Thank you.
Richard

[adrian.brock]

Most likely it is jsp compiles.

Precompile your jsps if you don't want to compile them
at runtime.

If you still want to use jsps make sure you turn off
development mode for the jasper servlet.
see jbossweb-tomcat41.sar/ for the configs

Regards,
Adrian

[cp]

Adrian,
Thank you for your suggestions. I have turned off the development mode in Jasper for the short term and we are coming up to speed on precompiling the application before deployment.

I did observe that the java processes became smaller after setting the Jasper "development" parameter to "false" (down to ~120M from ~200M), however I am still observing a slowdown (perhaps not as severe) that is accompanied by java processes dieing and being recreated.

Here is a further observation of the correlation between processes and slowdown:
- I browse a link on the site => after a lag, two java processes appear (122M, 13.9% MEM)
- I am then able to quickly hit several pages with immediate response => more java processes are created up to 5 or 6 (all at 122M, 13.9% MEM)
- after 10-20 seconds my page request will hang => meanwhile java processes are disappearing from top. Sometimes after all of the processes are gone, the page will return. Following that the slowdown always accompanies either the creation or destruction of java processes. But the longest lag always occurs when I have clicked on a link and then observe all of the java processes disappearing.

"ps ax" shows 62 java server processes

I am more familiar with Apache, it may help if I can understand whether the java processes are equivalent to httpd children? Is the server process visible, or am I just seeing invocations of the JVM? Is it normal for the processes to be over 100M?

Here is a sample output of top:
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
28044 root 9 0 129M 129M 28376 S 1.1 14.7 0:26 java
28064 root 9 0 129M 129M 28376 S 0.9 14.7 0:00 java
28045 root 9 0 129M 129M 28376 S 0.7 14.7 0:26 java
28067 root 9 0 129M 129M 28376 S 0.3 14.7 0:09 java

I will appreciate any further illumination you can provide.

Thank you.
Richard

[adrian.brock]

These are threads (can you see all the memory usage is the same
they are sharing the same memory).

Use pstree -p

You can see what each thread is doing if you hit
control-\ on the console or send kill -3 to the jboss pid
(warning this temporarily stops the jvm while it dumps the state)

Most likely it is compiling the jsps.

Regards,
Adrian

[darranl]

Is there any chance that this could be due to garbage collection being performed?

[adrian.brock]

I doubt the threads are due to garbage collection.
The GC takes over an existing thread/threads.

Regards,
Adrian