2 Replies Latest reply on Jan 18, 2016 5:07 PM by shauna

    Java MapReduceTask task manager.

    shauna

      Hello everyone,

       

      I see the 8.1.0.Final release introduced task management and implemented a scripting task manager (see https://issues.jboss.org/browse/ISPN-4956)

      Are there any plans to introduce an implementation that will allow existing mapreduce tasks to use the new task manager?

       

      So, for example, the following job from the weather tutorial

       

         public void computeCountryAverages() {

            System.out.println("---- Average country temperatures ----");

            MapReduceTask<String, LocationWeather, String, Float> countryTemperatureTask = new MapReduceTask<String, LocationWeather, String, Float>(cache);

            countryTemperatureTask.mappedWith(new CountryTemperatureMapper()).reducedWith(new CountryTemperatureReducer());

            Map<String, Float> averages = countryTemperatureTask.execute();

            for(Entry<String, Float> entry : averages.entrySet()) {

               System.out.printf("Average temperature in %s is %.1f° C\n", entry.getKey(), entry.getValue());

            }

         }

       

      can be run as a managed  task.

       

      Doing this would enable existing java m/r tasks to be more manageable.

       

      Thanks.

        • 1. Re: Java MapReduceTask task manager.
          nadirx

          Hi Shaun,

           

          MapReduceTask is on its way out, since the new Java 8-based streams API is much more powerful and flexible. Infinispan 8.2 will come with the ability to deploy tasks as JARs in server and monitor them via the management console.

           

          Tristan

          • 2. Re: Java MapReduceTask task manager.
            shauna

            Hi Tristan,

             

            Ok great thanks for clarifying this.

            I am now wondering if the streams api will allow the data within Infinispan to processed by Apache Spark's MLib (MLlib | Apache Spark).

            I'm aware another approach would be to use the Infinispan spark connector to achieve this but wondered it could be done directly so could be more efficient.

             

            Cheers,

             

            Shaun