Introduction

This blog post is the part of the series that showcase some of the work in JBeret family projects for modernizing Java batch processing.  Previously, we've discussed how to move your batch processing workload developed for Java SE standalone environment (see this post) and Java EE platform (see this post) to PaaS such as OpenShift.  They serve as good proof that standard-based batch applications can run with Java SE and Java EE (now Jakarta EE) both locally on bare meta and on cloud platform.

While it is great to be able to choose from either Java SE or Java EE environment for batch application, you may be wondering if it is possible to take advantage of both: leverage the lightweightness and flexibility of Java SE, and also the wide range of platform service in Java EE.  The answer is yes.  With the advent of Eclipse MicroProfile, the new opensource enterprise Java standard for microservices architecture, this type of usecase is well supported.  In this post, we will explore how to build and run microservices batch application with Thorntail (an implementation of Eclipse MicroProfile) both locally and on OpenShift.  We will be using numbers-chunk-thorntail, a fully functional sample batch application with Thorntail, to illustrate the steps.

Build and Run Thorntail-based Batch Application Locally

First, let's see how to build and run the sample application the traditionaly way locally, and familiarize ourselves with the application structure and batch job.  numbers-chunk-thorntail is a Java EE batch processing application and contains a single batch job as defined in numbers.xml.  Additionally, the project pom.xml declares Thorntail-related dependencies and plugins that enhance the regular WAR file into executable Thorntail-style uber jar (fat jar).

The numbers.xml batch job contains a single chunk-type step that reads a list of numbers by chunks and prints them to the console. The 2 batch artifacts used in this application are:

• arrayItemReader: implemented in jberet-support, reads a list of objects configured in job xml
• mockItemWriter: implemented in jberet-support, writes the output to the console or other destinations

This application also contains a singleton EJB, StartUpBean, which starts execution of batch job numbers.xml upon application deployment and Thorntail server start.

For complete batch job definition, see the JSL file numbers.xml.

To git-clone the sample application from github:

git clone https://github.com/jberet/numbers-chunk-thorntail.git 

To build the sample application with Maven:

mvn clean install  

The above step produces both a regular webapp WAR file, and an executable uber jar (fat jar) containing Thorntail runtime and all dependencies:

The above step produces both a regular webapp WAR file, and an executable uber jar (fat jar) containing

Thorntail runtime and all dependencies:

ls -l target/
-rw-r--r--  1 staff     258412 Sep 29 14:46 numbers-chunk-thorntail.war
-rw-r--r--  1 staff  112089497 Sep 29 14:46 numbers-chunk-thorntail-thorntail.jar

To run the Thorntail-based application locallym, you can either directly run it with the familiar java -jar command, or with mvn:

# Run with java -jar
java -jar target/numbers-chunk-thorntail-thorntail.jar

# Or run with mvn
mvn thorntail:run

The output should show that Thorntail is bootstrapped, batch application deployed, batch job started and completed, and Thorntail server keeps running.

Apart from this automatically started initial batch job execution, you can perform various batch processing operations with RESTful API calls, thanks to jberet-rest included in this application.  For instance, try the following commands in a separate terminal window:

# to start the job named `numbers`
curl -s -X POST -H 'Content-Type:application/json' "http://localhost:8080/api/jobs/numbers/start" | jq

# to get the details and status of the newly started job execution
curl -s "http://localhost:8080/api/jobexecutions/1" | jq

# to get all step executions belonging to this job execution
curl -s "http://localhost:8080/api/jobexecutions/1/stepexecutions" | jq

# to abandon the above job execution
curl -X POST -H 'Content-Type:application/json' "http://localhost:8080/api/jobexecutions/1/abandon"

# to schedule a job execution with initial delay of 1 minute and repeating with 60-minute interval
curl -s -X POST -H 'Content-Type:application/json' -d '{"jobName":"numbers", "initialDelay":1, "interval":60}' "http://localhost:8080/api/jobs/numbers/schedule" | jq

# to list all job schedules
curl -s "http://localhost:8080/api/schedules" | jq

# to cancel a job schedule
curl -s -X POST -H 'Content-Type:application/json' "http://localhost:8080/api/schedules/1/cancel" | jq

# to get details of a job schedule
curl -s "http://localhost:8080/api/schedules/2" | jq

In above commands, a utility program called jq is used to pretty-print the JSON output. Its usage here is equivalent to "python -m json.tool".  To shut down the application, press Ctrl-C in the terminal window of the running application.

Also worth mentioning is Thorntail Project Generator, which can be used to create the scaffold of your application.  It allows you to pick and choose which technologies and frameworks to use, and generates the required maven dependencies and plugins, and key application classes.  This comes handy for quickly starting new Thorntail-based development projects.

Build and Deploy Thorntail-based Batch Application to OpenShift Online

Next let's look at how to build and deploy the same batch application to OpenShift.  OpenShift will need to enlist a Java SE runtime, and here we choose to use openjdk18.  All the operations we will be performing can be done via either OpenShift command line tool (oc), or OpenShift Web Console.  For the sake of brevity, we will use oc commands.  For introduction to various features in OpenShift, you may want to check out OpenShift interactive tutorials.

We assume you already have an OpenShift account, and to log in:

oc login https:xxx.openshift.com --token=xxx  

To create a new project, if there is no existing projects:

oc new-project   

We wil use openjdk18-openshift image stream. Check if it is available in the current project:

oc get is  

If openjdk18-openshift is not present, import it:

oc import-image my-redhat-openjdk-18/openjdk18-openshift --from=registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift --confirm  

To create a new application (with default name):

oc new-app openjdk18-openshift~https://github.com/jberet/numbers-chunk-thorntail.git  

The above command will take a few minutes to complete, and to watch its status, run the following command:

oc rollout status dc/numbers-chunk-thorntail

To expose `numbers-chunk-thorntail` application to external clients, run the command:

oc expose svc numbers-chunk-thorntail

The above command exposes our batch application service to external client, by creating a route.  To get the route information, filtering by label -l option:

oc get route -l app=numbers-chunk-thorntail

Scale Thorntail-based Batch Application on OpenShift

OpenShift makes it very easy to scale up and down your deployments, through either command line or web console.  Let's first check what pods are running the batch application:

oc get pod -l app=numbers-chunk-thorntail

To view the application runtimne log in an text editor:

oc logs numbers-chunk-thorntail-1-nqnjj | view -

From the above log output, you can see that the application has been successfully built and deployed to OpenShift online, and the batch job has been started and completed.  As you can see, there is only 1 pod running this batch application.  To scale it up to 3 pods to service heavy load, and check the number of pods:

oc scale --replicas=3 dc numbers-chunk-thorntail
deploymentconfig.apps.openshift.io "numbers-chunk-thorntail" scaled

oc get pod -l app=numbers-chunk-thorntail

Among the 3 pods listed above, we can see that 2 are newly started, along with the one that has been running for a while.  Now let's check how to scale it back.  Say, we want to scale it back to 2 if the current replica count is 3:

oc scale --current-replicas=3 --replicas=2 dc numbers-chunk-thorntail
deploymentconfig.apps.openshift.io "numbers-chunk-thorntail" scaled

oc get pod -l app=numbers-chunk-thorntail

Access Thorntail-based Batch Application on OpenShift through REST API

Once the application is deployed to OpenShift, you can invokes its REST API to perform various batch processing operations. The steps are the same for both local Thorntail or OpenShift Thorntail runtime.

# to start the job named `numbers`
curl -s -X POST -H 'Content-Type:application/json' "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/jobs/numbers/start" | jq

# to get the details and status of the newly started job execution
curl -s "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/jobexecutions/1" | jq

# to get all step executions belonging to this job execution
curl -s "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/jobexecutions/1/stepexecutions" | jq

# to abandon the above job execution
curl -X POST -H 'Content-Type:application/json' "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/jobexecutions/1/abandon"

# to schedule a job execution with initial delay of 1 minute and repeating with 60-minute interval
curl -s -X POST -H 'Content-Type:application/json' -d '{"jobName":"numbers", "initialDelay":1, "interval":60}' "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/jobs/numbers/schedule" | jq

# to list all job schedules
curl -s "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/schedules" | jq

# to cancel a job schedule
curl -s -X POST -H 'Content-Type:application/json' "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/schedules/1/cancel" | jq

# to get details of a job schedule
curl -s "http://numbers-chunk-thorntail-pr.xxxx.xxxx.openshiftapps.com/api/schedules/2" | jq

Summary

We've just finished developing, deploying and running a batch processing microservices application based on Thorntail, in both local environment and OpenShift Online.  The combined power of Thorntail MicroProfile and OpenShift has opened up many possibilities for developing and managing batch processing applications.  I hope you find the information presented here useful, and as always, feedback and comments are much appreciated.