Recently we run into a serious problem with our JBoss 4.2.1 server. About an hour (sometimes shorter) after the server is started, many threads (hundreds of them) start frequent lockup, all waiting to acquire a "synchronized" lock on AppClassLoader. And the thread that holds the lock has the following stack trace:
-- snip --
I did a little timing exercise. This simple findBootstrapClass() call, which usually takes a few milliseconds at most, would take 20 seconds to multi-minutes to complete. During this time, since AppClassLoader.loadClass() is synchronized, the entire application simply stops and the requests quickly stack up and crash the application as a result.
I'm not saying this is a JBoss or JVM bug (though could be), since this only happens after we recently deploys our new application code. Some change in our code triggered this behavior but we don't know what it is, and this behavior only shows up in one of our JBoss servers.
Any experience on this kind of lockup? This problem has jeopardized our release schedule and any help will be greatly appreciated.