Usually in these cases, we obtain a thread dump, which lists all the current threads in jboss and the activities being carried out by them. Here's how it's done:
See if it helps.
Thanks for the response and apologies for my late reply.
The kill -3 worked fine and we managed to get the thread/stack trace.
Initially we had some problems relating the threads to the busy ones on the system. In many cases, the thread did not exist in the listing. However, by subtracting 1 from the LWP number the unix prstat command gave us, we could match the NID entry in the thread trace. This consistantly showed the same piece of code was causing the problem.
The final cause of the problem turned out to be the generated sql that was multiple rows from a table (instead of one). As the system has been used, this table has grown and we obviously hit a critical point and system performance dropped off.
We managed to reproduce this in our test environment and improved performance by flushing old entries from the table. The root cause still exists but at least we know how to manage it.
Thanks for you help