Running Batch Application in Cloud Foundry

Cloud Foundry has become a platform of choice for large enterprises to run cloud native applications and transform the way applications are built and deployed. It provides the agility, scalability and flexibility to continuously launch new features quickly while reducing the risk.

Cloud Foundry has been widely adopted to run Microservices which are associated with a long running process that never ends. Cloud Foundry as a platform monitors all the applications and ensures that the defines number of container instances are always running. It automatically deletes the failed instances and launches new instances without any manual intervention. The application instances are ephemeral i.e. containers running the application destroyed to be replaces with new instances if anything goes wrong. This approach works fine for scenarios where the process is expected to run continuously, however it won’t be applicable to batch applications.

Spring Cloud Task allows to create a short-lived microservices that can be run as a one-off task in Cloud Foundry. After the process is completed, the container running the task is destroyed to release the resources. This is quite powerful in the scenarios where heavy workload can be run without having any dedicated machines for running batch applications.

Pivotal Cloud Foundry

In order to demonstrate Spring Batch application running as a one-off task in Cloud Foundry, we can run the Pivotal Cloud Foundry (PCF) in a virtual machine locally. It consumes quite a bit of resources (8 GB RAM) while running locally, however provides an easy access to running applications in Cloud Foundry. Follow the link to download PCF Dev and the instructions on how to set-up Cloud Foundry running in your personal device.

Apps Manager can be accessed by navigating to the link https://apps.local.pcfdev.io

Spring Batch Application

The sample batch application discussed in this post is available on GitHub.

The batch application processes an input file containing payment transactions to add the merchant name in the output file. This is a very basic batch application with a single step containing a reader, processor and writer.

The processor looks for a merchant id in the input file and adds the corresponding merchant name.

public Transaction process(Transaction item) throws Exception {

if ("1001".equalsIgnoreCase(item.getMerchantId())) {
item.setMerchantName("Amazon");
} else if ("1002".equalsIgnoreCase(item.getMerchantId())) {
item.setMerchantName("Walmart");
} else {
item.setMerchantName("Not Available");
}
System.out.println("Enriched Trasaction Details --> " + item.toString());
return item;
}

The writer prints the enriched transaction record to the console.

public void write(List<? extends Transaction> items) throws Exception {
List<String> enrichedTxnList = new ArrayList<>();
items.forEach(item -> {
String enrichedTxn = String.join(",", item.getTransactionId(), item.getMerchantId(), item.getMerchantName(), item.getTransactionAmt());
enrichedTxnList.add(enrichedTxn);
});
enrichedTxnList.forEach(System.out::println);
//Files.write(Paths.get("./output/transaction-enriched.txt"), enrichedTxnList, StandardOpenOption.CREATE,StandardOpenOption.APPEND);
}

The batch is annotated to run as a Task application that can be executed in cloud foundry in its own container.

@SpringBootApplication
@EnableTask
@EnableBatchProcessing
public class PaymentProcessingBatch {

public static void main(String[] args) {
ApplicationContext ctx = SpringApplication.run(PaymentProcessingBatch.class, args);
}
}

manifest.yml file defines properties that needs to be applied while pushing the application to cloud foundry. Since this is a batch application we don’t want to create a route and define a hostname.

---
applications:
- name: payment-processing-batch
memory: 1G
random-route: true
path: build/libs/payment-processing-batch-0.0.1-SNAPSHOT.jar
no-hostname: true
no-route: true
health-check-type: none

Push the application to cloud foundry by executing following command in PCF CLI.

cf push -t 180

PCF will stage the application, create the droplet using the java buildpack and start a container instance to run the batch application task. Stop the application instance manually so that it can be run as a one-off task as needed.

Execute the following command to run the batch application as a task

cf run-task payment-processing-batch ".java-buildpack/open_jdk_jre/bin/java org.springframework.boot.loader.JarLauncher"

We can also see the executed tasks and their status by navigating to the tasks tab under the batch application in the PCF App Manager console. As we can see in the image below, batch application task was executed successfully in a short lived PCF container.