HudsonHowToDebug

    VisualVM

    You can start "jvisualvm" with jdk1.6.0_07 on any master or slave machine so you can attach to the java process.

    Hudson Master

    We are running Hudson under JBoss Enterprise Application Platform, thus before you start the AS you need to specify debug parameters:  export JAVA_OPTS=" -Xdebug -Xnoagent -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"  Then just simply point your debugging tool to <hudson_host>:8787. Hudson also has built-in debugging capabilities via Jetty plugin - see http://hudson.gotdns.com/wiki/display/HUDSON/Building+Hudson.

    Hudson Slave

    To debug this, you need to be able to attach a debugger to the slave, and that means launching slaves with the JPDA support on. For JNLP slaves unfortunately I don't know how to do this, but for slaves that are launched by the master, you can do this by adding something like "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=5005" to the command line.  If you see exceptions, be sure to capture that as it's helpful for us to see what the potential problem might have been.

    Reference a slave from the groovy console

     hudson.model.Hudson.instance.getComputer("dev91-linux").getExecutors()[4]
    println hudson.model.Hudson.instance.getSlave("dev91-linux").getLabelString()
    
    computer = hudson.model.Hudson.instance.getComputer("dev91-linux")
    
    //execute groovy command on a slave
    hudson.util.RemotingDiagnostics.executeGroovy("println \"hallo\"",computer.getChannel())
    
    //find out what's happening in the executor
    executor = computer.getExecutors()[EXECUTOR_INDEX]
    println ("Project: "+ executor.executable.getParent() )
    println("Stack trace: \n")
    stack = Thread.getAllStackTraces()[executor]
    for(i in stack) {
    println(i)
    }
    
    //who owns SCM lock that blocks the executor? (no null checks!)
      //if "java.lang.NullPointerException: Cannot get property: parent on null object" then we are not blocked by another SCM operation
      //if "Cannot invoke method getLock() on null object" - same as well
    executor.executable.parent.getTriggers().values().each {
       scmLock=it.getLock();
       println "SCM LOCK BY: " + scmLock.owner
       stack = Thread.getAllStackTraces()[scmLock.owner]
       for(i in stack) {
          println(" "+i)
       }
       // ... possibly kill the culprit: scmLock.owner.stop()
    }
    
    
    // reset dead executors
    execs = computer.executors
    for(e in execs) {
    if(!e.isAlive()) {
    computer.removeExecutor(e)
    }
    }
    computer.setNode( computer.getNode() );
    
    
    

     

     

    Some more handy groovy scripts

    get thread dump of master

    In groovy console execute:

    for(entry in Thread.getAllStackTraces().entrySet()) {
       println entry.key
       for(s in entry.value) println "    " + s
    }
    
    misc
    println hudson.model.Hudson.instance.getItem("JBoss-Metadata").workspace.channel
    t = hudson.model.Hudson.instance.getItem("JBoss-Metadata").getTriggers().get(hudson.triggers.SCMTrigger.DESCRIPTOR).getLock().owner
    println t
    stack = Thread.getAllStackTraces()[t]
    for(p in stack) {
    println("    "+p)
    }
    println "\n\n"
    
    for( i in hudson.triggers.SCMTrigger.DESCRIPTOR.getItemsBeingPolled()) {
       println "PROJECT: " + i
       println "WORKSPACE CHANNEL: "+i.workspace.channel
       def k = i.getTriggers().get(hudson.triggers.SCMTrigger.DESCRIPTOR).getLock().owner
       if(k != null) {
    println "THREAD: "+k
    stack2 = Thread.getAllStackTraces()[k]
    for(l in stack2) {
    println("    "+l)
    } 
       }
    }
    

     

    Find out which builds were executing during a given time range (builds should not have been deleted):

    import java.text.SimpleDateFormat;
    import java.util.GregorianCalendar;
    
    df=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
    starttime=new GregorianCalendar();
    endtime=new GregorianCalendar();
    buildEnd=new GregorianCalendar();
    
    //------- look here -------
    starttime.setTime(df.parse("2008-5-10 10:10:10"));
    endtime.setTime(df.parse("2008-5-10 11:10:10"));
    //------- user settings end -----
    
    for (hudson.model.AbstractProject p : hudson.model.Hudson.instance.projects) {
        for (hudson.model.AbstractBuild b : p.getBuilds() ) {
            if (b==null) break; //I think that can't be null anyway
            if ( b.isBuilding() || b.timestamp.compareTo(endtime) >= 0 ) continue;
            buildEnd.setTimeInMillis(b.timestamp.getTimeInMillis()+b.duration);
            if ( buildEnd.compareTo(starttime) < 0 ) break;
    
            println b;
        }
    }