My server gets jammed

[ericjava.eric.chiralsoftware.net]

Now I'm having a strange problem on my server that is quite serious.

I launched our website (chiralsoftware.com) using Seam 2.02SP1.  For whatever reason, it works fine for a while (a couple of hours?) and then it jams, with the CPU pegged at 100%.  The Java process that is JBoss is the one that does that.  The site gets some traffic but it should be easily able to handle it.

Any ideas on how to approach this?

Thanks

[ericjava.eric.chiralsoftware.net]

Some more info: I looked at the logs and saw logs like this:

11:02:48,669 INFO  [Contexts] starting up: org.jboss.seam.security.identity
11:02:48,669 INFO  [Contexts] starting up: org.jboss.seam.web.session
11:03:23,389 INFO  [Contexts] starting up: org.jboss.seam.security.identity
11:03:23,390 INFO  [Contexts] starting up: org.jboss.seam.web.session
11:03:40,500 INFO  [Contexts] starting up: org.jboss.seam.security.identity
11:03:40,500 INFO  [Contexts] starting up: org.jboss.seam.web.session
11:03:57,447 INFO  [Contexts] starting up: org.jboss.seam.security.identity
11:03:57,448 INFO  [Contexts] starting up: org.jboss.seam.web.session
11:04:14,234 INFO  [Contexts] starting up: org.jboss.seam.security.identity
11:04:14,235 INFO  [Contexts] starting up: org.jboss.seam.web.session
11:04:30,857 INFO  [Contexts] starting up: org.jboss.seam.security.identity

Those are hits from Googlebot.  Unfortunately we had a bunch of s:links on the site and so Googlebot got a bunch of cid=XXX links in its crawl-list.  I've set Googlebot to slow crawl, removed all the s:links, and used robots.txt to exclude it from some pages. 

It's starting a lot of sessions there.  Is this causing the problem?  Is there something I could / should do?

[dro_k]

Take a look at these

My Link

My Link

[ericjava.eric.chiralsoftware.net]

Thanks for the links.  I'm going into this in more depth because I do want users to have conversations active, but it sure did bring down the entire site when Googlebot got those URLs into its crawl queue.

So I wrote an article about Seam and Googlebot, going into detail of using URLRewrite to help with this problem.  I haven't done it yet, but I'm planning to turn propagation back on, and use a URLRewrite rule to strip CIDs out with an outbound rule.  I'll post about that.  I think people have also used URLrewrite to strip out jsessionids from Googlebot, which is a similar type of usage.

My solution using URLRewrite is working nicely and the site is back to normal now.

[ericjava.eric.chiralsoftware.net]

I spoke too quickly.  I just saw another out of memory error.  Something else is causing this.

[felipealbertao]

Have you tried to increase the VM memory on run.bat or run.conf? Also, are you using a Virtual Server? (Xen, Virtuozzo, etc...) I am asking this question because I experienced a similar situation a while ago when the server usage was stuck at 100%, when in fact the issue was the the VPS didn't have enough memory -- I imagine in my case the 100% usage was the memory swapping due to lack of memory.

[ericjava.eric.chiralsoftware.net]

This is plain old Java 6 on a Suse 10.1 machine with lots of memory.  No virtualization.

I already did increase the memory:

JAVA_OPTS="-Xms256m -Xmx1024m
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000"

Should I go higher?  Could this really be the problem?  This server isn't getting such high levels of traffic. Can it really consume a gigabyte to load a blog entry from the database every couple of minutes?

[felipealbertao]

You're correct... In fact, my production server is running with half of your maximum (-Xmx512m). I really cannot understand why this is happening with your app. We have stress-tested multiple releases of our Seam-based app, and all I can say is that Seam definitely handles more load than that! FYI, I'm using Seam 2.0.1.GA, JBoss 4.2.2.GA and JRockIt 1.5 on a Amazon EC2 virtual server (RedHat image)

I'd try to first isolate the problem: I. e.: Is it something related to your server, VM, the framework, or the app itself? I'd start by downgrading Seam to 2.0.1.GA. Then I'd try another JDK (perhaps Sun's 1.5 or BEA JRockIt). Then, I'd try another physical server. If nothing works, then at least you isolated the problem to the app itself...

Also, you might find JMX Console's ServerInfo useful.

Felipe

[ericjava.eric.chiralsoftware.net]

Felipe, thanks for the suggestions.  I'll give them a try, but my first is going to be to run the server with -XX:+HeapDumpOnOutOfMemoryError .  I can then run the dump through various dump analyzers to see if I can get some clues from that.

It could very well be the application.  It's nothing strange of an app, but it does do some crypto stuff, etc.  It could be anywhere and a dump analyzer might show me.

The app certainly is performing better now that I've used URLRewrite to block GoogleBot from loading CID URLs.  But it should never give an out of memory.

I know, Seam and JBoss AS are stable things.  I'm guessing 99% it's in my app somewhere, or in some interaction with an external resource (Postgres).

[dro_k]

I just read your post. Very interesting and job well done. The only thing I noticed was that you're getting

java.lang.OutOfMemoryError: PermGen space

which I don't think are necessarily caused by Google Bot. If the problem was with the big number of sessions/conversations then IMO you should get OutOfMemoryError for Heap space not the PermGen space. To my knowledge PermGen is where Java stores meta-data about loaded/created classed, and is a common error while development mainly caused by HotDeployment of classes. Are you sure you don't have an actual memory leak?

cheers,

Drew

[ericjava.eric.chiralsoftware.net]

Drew Kutcharian wrote on Jun 01, 2008 03:50:


I just read your post. Very interesting and job well done.

Thank you!  I always post my investigation, hoping that it helps others.  And, in fact, we get constant visitors with search terms that are related to problems we have also run into.

The only thing I noticed was that you're getting

java.lang.OutOfMemoryError: PermGen space

which I don't think are necessarily caused by Google Bot.

Right, at this point, I think Googlebot might not have been the factor.  However, that problem with Googlebot needed to be solved.  I do not want Googlebot finding 1000 duplicate pages of every page on my site, and I don't want it making 5,000 page requests a day on the site either.  Longer term, we're going to use the awesome power of Seam to keep on adding more aspects to the site that adjust to the user's experience.  We need to make sure we can do that in a way that doesn't create a spider trap.

If the problem was with the big number of sessions/conversations then IMO you should get OutOfMemoryError for Heap space not the PermGen space. To my knowledge PermGen is where Java stores meta-data about loaded/created classed, and is a common error while development mainly caused by HotDeployment of classes. Are you sure you don't have an actual memory leak?

I'm not sure of anything on this, other than, the site goes down after an hour or so, with moderate traffic which should be absolutely no problem for it.

Is there anything else I should look at?  I'm running these things with debug on.  Is that the fatal problem?

[lucdew.luc.dewavrin.gmail.com]

Yes,
maybe you should just increase perm size (-XX:PermSize and -XX:MaxPermSize) due to a high number of loaded classes , the Sun JVM default values are quite low (for Linux 32 bits JVM 1.6 in server mode i think it's 16M and 64M Bytes respectively ).
Also you might use the concurrent mark and sweep GC, to clean permanent space (especially your app run on multicore machine)

See for JVM options
http://blogs.sun.com/watt/resource/jvm-options-list.html
And out of memory issues related to permanent space:
http://my.opera.com/karmazilla/blog/2007/03/13/good-riddance-permgen-outofmemoryerror

Using a profiler (like JProfiler) in a test environment would really help to spot the issues.

Anyway, thanks for the very interesting blog post about Google bot and cid.

Luc

[felipealbertao]

Just for the record: I used to get a lot of OOM exceptions with Sun JDK 1.5 after a certain number of ear redeployments. The solution as pointed by Drew and Luc is to increase the perm size, but I also noticed that this is not necessary when using JRockIt. Apparently JRockIt does a much better job of memory management than Sun JDK. It's understandable, since JRockIt was designed from the ground-up for server deployments. Unfortunately, it looks like that JRockIt is not available for download anymore after Oracle bought BEA... It's a bummer, because it is really a great VM for server-side deployments. The last JRockIt 5 release is still available at download.com, though.

[kukeltje.ronald.jbpm.org]

Apparently JRockIt does a much better job of memory management than Sun JDK. It's understandable, since JRockIt was designed from the ground-up for server deployments.

afaik, it shares ~95% of the code with the SUN JVM

Unfortunately, it looks like that JRockIt is not available for download anymore after Oracle bought BEA...

What's next? WLS? ;-)

It's a bummer, because it is really a great VM for server-side deployments.

Hmmmm... true to a certain extend, but we experienced other (negative) issues that did not occur with the SUN JVM

[ericjava.eric.chiralsoftware.net]

Thanks for the suggestion re: PermGen.  That was definitely the problem.

I wanted to really understand what was going on with this, so I did a bunch more tests to determine how much PermGen space is needed per Seam application.  I came up with a formula for it, and revised my post.  Check for the formula for calculating PermGen space for Seam apps.

Bottom line: JBoss AS deployment that has more than 4 Seam apps deployed on it WILL encounter PermGen errors.  I derived a formula to use for capacity planning of this; see the article.