"bstansberry@jboss.com" wrote:
So are you thinking about a service that generates thread dumps (or tweaking the existing one), with configuration options to control where the dumps go?

"bstansberry@jboss.com" wrote:
With hooks to inject the service into other interested services, e.g. HAPartition? HAPartition would then decide whether an event (e.g. a timeout on an RPC) justifies calling into the thread dump service.

In the case of an RPC timeout, only the caller knows it happened.

Couple of things on Thread.getAllStackTraces()

It leaks :). Fixed in 6u1
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6434648

From: http://wiki.jboss.org/wiki/Wiki.jsp?page=StackTrace

This function returns the information provided by Thread.getAllStackTraces(). This does not include the light weight thread id, which is important for troubleshooting hung and spinning processes; therefore, it is recommended to use the methods described above to obtain the stack trace if possible.

There're 4 ways to generate a thread dump. I have taken the output from the same run with different methods:

1.- kill -3. Can be parsed by tools like https://tda.dev.java.net/ (simply, one of the most useful tools I've found out there!). Example:

"JBossLifeThread" prio=1 tid=0x0a690cc0 nid=0x1741 in Object.wait() [0x867f2000..0x867f30b0]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x9247e200> (a java.lang.Object)
 at java.lang.Object.wait(Object.java:474)
 at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)
 - locked <0x9247e200> (a java.lang.Object)

Thread: JBossLifeThread : priority:5, demon:false, threadId:58, threadState:WAITING, lockName:java.lang.Object@10cff6b

java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)

Name: JBossLifeThread
State: WAITING on java.lang.Object@10cff6b
Total blocked: 0 Total waited: 1

Stack trace:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:940)

http://jira.jboss.com/jira/browse/JBAS-5163 for the thread dump discussion.

I'll create a JIRA for the other topic asap.

I've just realised that I never created a JIRA for a service that would request thread dumps from 1..n nodes upon some trigger or event.

Couple of weeks back I saw this JIRA which seems to attempt to do a similar thing: get a thread dump when the server shutdown times out.

https://jira.jboss.org/jira/browse/JBASM-3

The idea is basically the same except that the triggers are different. My initial motivation is members that didn't response to a sync call, whether AS cluster wide RPC, or JBC's repl sync of data.

I finally created a JIRA for this:

https://jira.jboss.org/jira/browse/JBAS-6311

Probe dump would indeed be really handy for trying to retrieve the thread dumps accross the cluster but it would be called from some Java logic (notification consumer for each different type of situation when to generate a thread) and hence I would need to call JGroups API to get the same that "probe.sh dump" would return.

Also, I would expect all thread dumps from problem X to be dumped locally and then someone to inspect them with TDA or Samurai, so string sounds good to me. Note that it would help that the thread dump string is formatted in the same way as the JVM does (kill -3) as this would help compatibility with tools that are already able to read these type of dumps.

Hmmm, I don't understand this very well. I need to test it but ideally what you want is that whatever API you call to get thread dumps for all nodes in the cluster returns in an array or similar each individual thread dump with information where the thread dump is coming from.

The idea here is that the problem plus each individual node's thread dumps are logged somewhere. Obviously, if the same problem happen a lot of times in a row, you'd only log it once, otherwise you could end up swamping the server's HD.

I'm not sure we'd want thread dump responses from different clusters at this point unless there was some kind inter cluster sync RPC call or replication.