VisualVM

You can start "jvisualvm" with jdk1.6.0_07 on any master or slave machine so you can attach to the java process.

Hudson Master

We are running Hudson under JBoss Enterprise Application Platform, thus before you start the AS you need to specify debug parameters:  export JAVA_OPTS=" -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"  Then just simply point your debugging tool to <hudson_host>:8787. Hudson also has built-in debugging capabilities via Jetty plugin - see http://hudson.gotdns.com/wiki/display/HUDSON/Building+Hudson.

Hudson Slave

To debug this, you need to be able to attach a debugger to the slave, and that means launching slaves with the JPDA support on. For JNLP slaves unfortunately I don't know how to do this, but for slaves that are launched by the master, you can do this by adding something like "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=5005" to the command line.  If you see exceptions, be sure to capture that as it's helpful for us to see what the potential problem might have been.

Reference a slave from the groovy console

 hudson.model.Hudson.instance.getComputer("dev91-linux").getExecutors()[4]
println hudson.model.Hudson.instance.getSlave("dev91-linux").getLabelString()

computer = hudson.model.Hudson.instance.getComputer("dev91-linux")

//execute groovy command on a slave
hudson.util.RemotingDiagnostics.executeGroovy("println \"hallo\"",computer.getChannel())

//find out what's happening in the executor
executor = computer.getExecutors()[EXECUTOR_INDEX]
println ("Project: "+ executor.executable.getParent() )
println("Stack trace: \n")
stack = Thread.getAllStackTraces()[executor]
for(i in stack) {
println(i)
}

//who owns SCM lock that blocks the executor? (no null checks!)
  //if "java.lang.NullPointerException: Cannot get property: parent on null object" then we are not blocked by another SCM operation
  //if "Cannot invoke method getLock() on null object" - same as well
executor.executable.parent.getTriggers().values().each {
   scmLock=it.getLock();
   println "SCM LOCK BY: " + scmLock.owner
   stack = Thread.getAllStackTraces()[scmLock.owner]
   for(i in stack) {
      println(" "+i)
   }
   // ... possibly kill the culprit: scmLock.owner.stop()
}


// reset dead executors
execs = computer.executors
for(e in execs) {
if(!e.isAlive()) {
computer.removeExecutor(e)
}
}
computer.setNode( computer.getNode() );

Some more handy groovy scripts

get thread dump of master

In groovy console execute:

for(entry in Thread.getAllStackTraces().entrySet()) {
   println entry.key
   for(s in entry.value) println "    " + s
}

misc

println hudson.model.Hudson.instance.getItem("JBoss-Metadata").workspace.channel
t = hudson.model.Hudson.instance.getItem("JBoss-Metadata").getTriggers().get(hudson.triggers.SCMTrigger.DESCRIPTOR).getLock().owner
println t
stack = Thread.getAllStackTraces()[t]
for(p in stack) {
println("    "+p)
}
println "\n\n"

for( i in hudson.triggers.SCMTrigger.DESCRIPTOR.getItemsBeingPolled()) {
   println "PROJECT: " + i
   println "WORKSPACE CHANNEL: "+i.workspace.channel
   def k = i.getTriggers().get(hudson.triggers.SCMTrigger.DESCRIPTOR).getLock().owner
   if(k != null) {
println "THREAD: "+k
stack2 = Thread.getAllStackTraces()[k]
for(l in stack2) {
println("    "+l)
} 
   }
}

Find out which builds were executing during a given time range (builds should not have been deleted):

import java.text.SimpleDateFormat;
import java.util.GregorianCalendar;

df=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
starttime=new GregorianCalendar();
endtime=new GregorianCalendar();
buildEnd=new GregorianCalendar();

//------- look here -------
starttime.setTime(df.parse("2008-5-10 10:10:10"));
endtime.setTime(df.parse("2008-5-10 11:10:10"));
//------- user settings end -----

for (hudson.model.AbstractProject p : hudson.model.Hudson.instance.projects) {
    for (hudson.model.AbstractBuild b : p.getBuilds() ) {
        if (b==null) break; //I think that can't be null anyway
        if ( b.isBuilding() || b.timestamp.compareTo(endtime) >= 0 ) continue;
        buildEnd.setTimeInMillis(b.timestamp.getTimeInMillis()+b.duration);
        if ( buildEnd.compareTo(starttime) < 0 ) break;

        println b;
    }
}