performance logging in tomcat or mod_jk

[jimyliu]

Hi,

We recently had JVM crash problems on tomcat server. But there is no hs_error or core dump file.  We tried to identify what URL request caused the JVM crash. But in the tomcat or mod_jk.log, it only shows the request entry when the request is finished. Is there any way I can see the request entry before when the request starts either in apache, mod_jk or tomcat local_access log ?

Thanks,

Jimy

[vcorrea]

Hi Jimy,

I would sugest that you analyse it with an jvm introspect tool. Im using byteman ( jboss.org/byteman ). You can write simple script to analyse when a class is istantiated. So if you have suspicious classes you can check its status on runtime. Hope that it can help

[jimyliu]

Hi Victor,   Thanks for the suggestion. I will test with byteman to see if that can help my situation.

[rhusar]

Jimy, you can configure your mod_jk to log more and you will see what was happening with the request (the logging will be however too verbose so not a good fit for production for sure).

If there are chances you can reproduce than surely +1 for Byteman if you want to instrument a class/method to do additional logging.

BTW "But there is no hs_error" - did the JVM user have permission to write it there? Make sure you are not calling System.exit() ;-)

[vcorrea]

Did you activated this option on your vm args?

XX:+HeapDumpOnOutOfMemoryError

?

[jimyliu]

Yes, configure the mod_jk in debug mode is too heavy and not possible for the live system.   We did try to examine our code to make sure there is no System.exit().  

Since there is frequently GC activities before it crash and the heap size looked ok, we did not set this option XX:+HeapDumpOnOutOfMemoryError , either.  But I will set it to see if that helps.

Thanks ,

Jimy

[rhusar]

So this issue happens relatively often in production env but you cant reproduce it? Ah, these are always hard to find in live system. Most likely this looks like memory issue, so make sure your system and your JVM are not running out of memory.

[jimyliu]

Sorry, I type incorrectly, there is NO frequently GC activities before it crash.  And yes, not reproducible either. :-(

Will definitely try the byteman if I can get the breaking point.