AWS Batch Job Scheduling Using AWS EventBridge

An optimise cost based approach compared with AWS ECS recurring cost

Cost optimisation is goal of every organisation especially in today’s day and age. But sometimes we do have batch job which is required run in Container mode either in ECS or EKS.

Below is the approach which I’ve worked on my recent assignment for reducing AWS ECS recurring cost for batch processes.

Assumptions :

1. I’m assuming here that we just have to process the Batch and we don’t provide any endpoint for on-demand execution of the task within the same service.
2. In AWS we do have role which has all the required permission and policies for this demo (Access related to AmazonECSTaskExecutionRolePolicy, EventBridge, Cloudwatch, etc.).
3. For this demo, I’ve created a public AWS ECR which I’ll be using in AWS Batch job.

Code Walkthrough :

Below is the sample code which I’ve written in order to represent our batch task.

Now, one point we should note that, I’ve also configured an aspect for graceful shutdown of the service as soon as our job is processed.

I’ve also used Makefile here in order to build and push our code to AWS ECR.
Code example can be found here.

Architecture Design :

On a high level, we’ll basically build the docker image of the code and push it to ECR which later on we’ll configure in AWS Batch. Once we have configured AWS job and tested it then we can schedule this job using AWS EventBridge.

AWS Batch + AWS EventBridge :

A brief intro about this service can be found in below references as well :

Understanding AWS Batch: A Brief Introduction and Sample Project | by Josh Kahn | Medium
AWS Batch — Easy and Efficient Batch Computing Capabilities — AWS (amazon.com)

: Setting up AWS Batch :

I’ve created a short video which shows the configuration of aws batch for this sample project.

https://www.dropbox.com/s/dl67ek3zww7yp5f/Batch.mp4

We need to provide default aws log driver so that our batch job logs will be available in cloud watch.

: Setting up AWS EventBridge :

Once we made sure that AWS batch job is running fine we now have to create a EventBridge with the resource ARN of the job queue & job definition along with our desired cron expression or duration.

https://www.dropbox.com/s/l7y0qb73kw8irpe/EventBridge.mp4

Now, this process can be automated further with integration of Terraform for deployment.

Note : Both the Batch service and EventBridge role, I’ve selected to be default.

Cost Estimation :

(for 1GB CPU & 2GB RAM for us-east-1 region)

ECS :
Unit conversions management events

• Number of tasks or pods: 1 per day * (730 hours in a month / 24 hours in a day) = 30.42 per month
• Average duration: 1 days = 24 hours

Pricing calculations

• 30.42 tasks x 1 vCPU x 24 hours x 0.03238 USD per hour = 23.64 USD for vCPU hours
• 30.42 tasks x 2.00 GB x 24 hours x 0.00356 USD per GB per hour = 5.20 USD for GB hours
• 20 GB — 20 GB (no additional charge) = 0.00 GB billable ephemeral storage per task
• 23.64 USD for vCPU hours + 5.20 USD for GB hours = 28.84 USD total
• Fargate cost (monthly): 28.84 USD

Batch + EventBridge :

As per aws documentation AWS Batch Pricing (amazon.com)

There is no additional charge for AWS Batch. You pay for AWS resources (e.g. EC2 instances, AWS Lambda functions or AWS Fargate) you create to store and run your application. You can use your Reserved Instances, Savings Plan, EC2 Spot Instances, and Fargate with AWS Batch by specifying your compute-type requirements when setting up your AWS Batch compute environments. Discounts will be applied at billing time.

So if we assume our job will run for 3 hours then the cost estimate would be :
Unit conversions management events

• Number of tasks or pods: 1 per day * (730 hours in a month / 24 hours in a day) = 30.42 per month

Pricing calculations

• 30.42 tasks x 1 vCPU x 3 hours x 0.03238 USD per hour = 2.95 USD for vCPU hours
• 30.42 tasks x 2.00 GB x 3 hours x 0.00356 USD per GB per hour = 0.65 USD for GB hours
• 20 GB — 20 GB (no additional charge) = 0.00 GB billable ephemeral storage per task
• 2.95 USD for vCPU hours + 0.65 USD for GB hours = 3.60 USD total
• Fargate cost (monthly): 3.60 USD

And as per eventBridge documentation Amazon EventBridge Pricing | Event Bus | Amazon Web Services cost which will be for us is $1.00/million custom events published.

So as a net result we can clearly see a significant portion price can be saved by implementation of this approach.

Thank you for reading & watching !!