1 Reply Latest reply on Dec 3, 2003 12:03 AM by jonlee

Tomcat / JBoss slowing to mysterious crawl over time

astehman Dec 2, 2003 3:56 PM

Greetings all,

We're running RH 9 with jboss 3.2.1 and integrated Tomcat. We are using the AJP connector to load balance from 2 apache www servers to 2 jboss instances (1 per node) in a 2-node cluster.

The servers for no apparent reason get very sluggish after a couple hours of running. We have two ideas, one pertaining to socket scarcity, the second to multiple instances of huge classes being maintained by classloader.

The first hypothesis is that there may not be sockets to go around. Example:

A user attempts to log in. The browser spins for a long time. Maybe five minutes later, the page returns with the user logged in. Sometimes this takes ten minutes.

Note this log entry:

218189 07:11:55,506 DEBUG [ChannelSocket] receive()
218190 07:12:15,513 INFO [ChannelSocket] connection timeout reached
218191 07:23:56,701 DEBUG [ChannelSocket] Accepted socket
Socket[*deleted for security purposes*]
218192 07:23:56,701 DEBUG [ChannelSocket] receive()
218193 07:23:56,701 DEBUG [ChannelSocket] read()

Entries such as on line 218190 ALWAYS precede entries such as on 218191. Note the time difference of over 11 minutes. Sometimes this is how long it takes for any HTTP requests to be processed (from the time the user clicks a button to the time the next screen is returned).

An additional note: we deployed 2 apps on each jboss instance. One is the full-blown app which uses scads of sockets for FTP and DB access, and also uses struts. The second is "hello world" JSP with a "hello world" session bean that returns the string (no ftp, no db access, no struts). The production app gets sluggish over time, while the "hello world" app does not. We even disabled the component which heavily utilizes ftp so this was not occurring, and the servers continue to slow down. This is observed by the testers who report increasingly long waits for page renders (all JSPs).

Another note is the production app makes use of a class that is over 10,000 lines long (not by our choice). There surely are multiple instances of this class being loaded and maintained by the classloader.

Many thanks for any insight you may have!

Alexandra

1. Re: Tomcat / JBoss slowing to mysterious crawl over time

jonlee Dec 3, 2003 12:03 AM (in response to astehman)

I'm not convinced the problem is with sockets. You aren't getting a problem with the other web app in the same VM/servlet container.

I would look at any issues with JDBC and perhaps start looking at any information you can get out of the JVM - garbage collection, memory profile. Try jmemprof (freeware) or Jprobe to help.

But probably do the simple thing first to debug where the slowdowns are occurring - some System.out.println or log4j statements would be useful. Possible areas are definitely execution of JDBC SQL, JMS if you are using it, and GC. But it is not limited to these areas.
Actions