3 Replies Latest reply on Apr 13, 2013 5:04 PM by jignaasa

    Distributed Execution - monitoring/tracking tasks in the various nodes

    jignaasa

      Hi,

       

      When running the PiApproximationDemo shipped with infinispan 5.1.6, after the call to DistributedExecutorService.submitEverywhere,

      is there a way to monitor (via JMX) the lifecycle of the tasks/Callables as they progress through the worker nodes and return back.

       

      Reason:

      1. We will be using a similar topology (ie. using infinispan as a computing grid) on a cluster with 100s of nodes and so would

      like to be able to centrally track/monitor the various tasks, as they go from master to worker nodes and return.

       

      Some of the metrics we are interested (in the master and each of the worker nodes):

      1. A list of failed Callables since node startup.

      2. In each of the worker nodes, details about Callables that have been submitted to the worker nodes and:

      * not yet started.

      * in-progress

      * Completed with a failure.

       

      I currently do not see these type of metrics published via JMX. Are there any plans of providing this level of instrumentation in the future.

      Could you please suggest the list of classes that would need to be extended for instrumentation would be helpful.

       

      Thanks,

      J

        • 1. Re: Distributed Execution - monitoring/tracking tasks in the various nodes
          vblagojevic

          J,

           

          There are plans for these features. Summarize your requirements and we'll see to include them in.

           

          Regards,

          Vladimir

          • 2. Re: Distributed Execution - monitoring/tracking tasks in the various nodes
            jignaasa

            Sorry for the delay in responding.

             

            Currently we are interested in a MBean in each of the Mapper/Worker nodes,

            that can have an interface like the one below.

            (We may also need similar instrumentation in the Master node: Basically get a list of Callables that have been submitted

            via the DistributedExecutorService.submitXXX methods, but have not yet completed / completed with failures)

             

            /**

            1. We would also like a configurable enableStatistics to turn these ON/OFF

            (Similar to the feature in for example Statistics: http://docs.jboss.org/infinispan/5.1/apidocs/jmxComponents.html#Statistics

            2. Note: All Callables below correspond to the DistributedExecuteCommand

            */

            public interface DistExecM/MXBean {

             

                      //Get a list of Callables that have been submitted to the Mapper but not started (since server restart)

                      Collection<Callable<?>> getTasksThatHaveNotStarted();

             

                      //Get a list of Callables that are currently running in the node

                      Collection<Callable<?>> getInProgressTasks();

             

                      //Get a list of failed Callables/DistributedCallables since server restart

                      Collection<Callable<?>> getFailedTasks();

             

                      void resetInProgressStatistics();

                      void resetFailedStatistics();

            }

             

            Please let me know if you need any more details.

             

            FYI, I tried to do this on my own, by writing a custom interceptor MBean (similar to the Statistics MBean above)

             

            import org.infinispan.interceptors.base.JmxStatsCommandInterceptor;

            import org.infinispan.jmx.annotations.MBean;

            ...

            ...

            MBean(objectName = "DistExec", description = "Distributed Execute Command details")

            public class JMXDistributedCommandInterceptor extends

            JmxStatsCommandInterceptor {

            ...

            ...

            @Override

            public Object visitDistributedExecuteCommand(InvocationContext ctx,

            DistributedExecuteCommand<?> command) throws Throwable {

             

            System.out.println("Started dist command " + command);

            inProgress.put(command, Boolean.TRUE);

            Object o = super.visitDistributedExecuteCommand(ctx, command);

             

            System.out.println("completed dist command " + command);

            inProgress.remove(command);

            return o;

            }

             

            The interceptor got activated but it was not showing up as MBean in jconsole.

            Upon looking into the source code, I found this is because infinispan scans for MBean annotations only at build time (and captures the information in: infinispan-core-component-metadata.dat, which is then used by at runtime by ComponentMetadataRepo

            to actually determine which MBeans are manageable and should be registered with the MBean Server)

             

            So the question is: Is there any way to build my own custom interceptor MBeans, that will automatically be picked up by infinispan and registered with the MBean Server, so it is visible with jconsole etc.

             

            (I cannot use JmxUtil.registerMBean since the custom interceptor is actually instantiated by infinispan)


            • 3. Re: Distributed Execution - monitoring/tracking tasks in the various nodes
              jignaasa

              For the custom interceptor MBean, I was initially trying to mimic the built-in interceptor MBeans packaged with infinispan which did not work.

              It works when my custom interceptor:

              1. extends org.infinispan.interceptors.base.BaseCustomInterceptor (as documented in the user guide.)
              2. Does not use the infinispan JMX annotations (Mbean etc.)  
              3. implements an interface XXMXBean (follows the JMX MXBeans approach)
              4. Overrides the start and stop methods from BaseCustomInterceptor and manually registers and unregisters itself from the Mbean server in these methods

               

              Minor caveats

              1. I cannot extend (and reuse) JmxStatsCommandInterceptor, but would instead have to implement: JmxStatisticsExposer. This is not such a big deal since JmxStatsCommandInterceptor does not have much code anyway.
              2. This interceptor MBean will be in a separate JMX domain from org.infinispan (where all the Cache Manager stuff is), which is probably how it should be, since it is custom interceptor and not part of org.infinispan

               

              Thanks,

              J