Orchestrating an application process with AWS Batch using AWS CDK : idk.dev


In lots of actual work purposes, you should use customized Docker photographs with AWS Batch and AWS Cloud Improvement Equipment(CDK) to execute advanced jobs effectively. AWS CDK is an open supply software program improvement framework to mannequin and provision your cloud utility assets utilizing acquainted programming languages, together with TypeScript, JavaScript, Python, C# and Java. For the answer on this weblog, we’ll use C# for the infrastructure code. It is a sequel to an earlier printed weblog with identical infrastructure and idea. On this weblog, we’ll leverage the capabilities and options of AWS CDK (with Microsoft .NET utilizing C#) as a substitute of CloudFormation. Let’s get began!

This publish supplies a file processing implementation utilizing Docker photographs and Amazon S3, AWS Lambda, Amazon DynamoDB, and AWS Batch. On this state of affairs, the consumer uploads a CSV file to into an Amazon S3 bucket, which is processed by AWS Batch as a job. These jobs will be packaged as Docker containers and are executed utilizing Amazon EC2 and Amazon ECS. The next steps present an outline of this implementation:

  1. AWS CDK bootstraps and deploys a CloudFormation template.
  2. The infrastructure launches the S3 bucket that shops the CSV recordsdata. Different AWS companies required for this utility orchestration are additionally spun up.
  3. The Amazon S3 file occasion notification executes an AWS Lambda perform that begins an AWS Batch job.
  4. AWS Batch executes the job as a Docker container.
  5. A Python-based program reads the contents of the S3 bucket, parses every row, and updates an Amazon DynamoDB desk.
  6.  Amazon DynamoDB shops every processed row from the CSV.


The next steps define this walk-through. Detailed steps are given all through the remainder of this publish.

  • Obtain the code and run the Microsoft Dotnet construct and AWS CDK deployment instructions (instructions offered under) to create the required infrastructure.
  • Drop the CSV into the S3 bucket (offered a pattern Pattern.csv that shall be parsed by the Batch Operation).
  • Affirm that the job runs and performs the operation based mostly on the pushed container picture. The job parses the CSV file and provides every row into DynamoDB.

As soon as the code is downloaded, please take a second to see how CDK supplies an easier implementation for spinning up an infrastructure utilizing C# code. You might use Visible Studio Code or your favourite alternative of IDE to open the folder (aws-netcore-app-using-batch). The offered supply code consists of the next main parts. Be aware the corresponding downloaded path for these parts offered under.

  • Microsoft .NET AWS CDK to construct the infrastructure (aws-netcore-app-using-batch/code/src/MyApp)
  • Python Lambda code as a part of assets for CDK to construct (aws-netcore-app-using-batch/code/assets)
  • Python Batch processor code and Dockerfile for the AWS Batch to execute (aws-netcore-app-using-batch/code/src/BatchProcessor)

Open the file “/aws-netcore-app-using-batch/cdk/src/MyApp/SrcStack.cs”. Code under (offered a snippet from the github resolution) spins up a VPC for the required CIDR and variety of availability zones.

IVpc todoVpc = new Vpc(this, Constants.VPC_NAME, new VpcProps{
    Cidr = Constants.CIDR_RANGE,
    MaxAzs = 4

Equally, builders have an possibility of implementing the CDK Constructs and might outline & deploy an AWS Service. Within the under snippet, Batch service function is created by implementing the assemble and returns an IAM Function. See full supply in /aws-netcore-app-using-batch/cdk/src/MyApp/Modules/BatchServiceRole.cs

public sealed class BatchServiceRole: Assemble {
    public BatchServiceRole(Assemble scope, string id): base(scope, id){
      this.scope = scope;
      this.id = id;
      this.Function = GetRole(

Factors to think about

  • The offered pattern makes use of Microsoft .NET C# AWS CDK as a substitute of CloudFormation. Some other supported programming language for CDK can be utilized. Please refer AWS CDK Documentation.
  • The pattern offered has Python code that’s packaged for Lambda Operate. This may be coded in another out there programming language.
  • For the AWS Batch Processing, a Python code & Dockerfile is offered utilizing which ImageAssets are constructed by the CDK. Container Picture is packaged and pushed as a part of CDK instructions execution. This Python code is run from the ECR which finally parses S3 file and pushes it to Dynamo. Supplied pattern is making utilizing of python. This may be coded in another out there programming language which permits code to be containerized.
  • As a part of this walkthrough, you employ the Optimum Cases for the batch. The a1.medium occasion is a cheaper occasion sort launched for batch operations, however you should use any AWS Batch succesful occasion sort in accordance with your wants.
  • To deal with the next quantity of CSV file contents, you are able to do multithreaded or multiprocessing programming to enrich the AWS Batch efficiency scale.
  • .NET Core 3.1, AWS CDK model 1.32.2, developer preview for .NET are used (out there on the time of writing). Newer implementation, different functionalities could also be out there with upcoming newer variations. Please be careful for newer releases which can take away any out of date implementation

When deployed, the AWS CDK creates the next infrastructure.

Orchestrating an Application using Batch

You’ll be able to obtain the supply from the github. Beneath steps will element utilizing the downloaded code. This has the supply code for AWS CDK that spins up the infrastructure. Moreover a lambda Python code and batch python utility (.py file) are offered. A pattern CSV file is offered which shall be processed by the batch. You’ll be able to optionally use the under git command to clone the repository as under. This turns into your SOUCE_REPOSITORY

$ git clone https://github.com/aws-samples/aws-netcore-app-using-batch
$ cd aws-netcore-app-using-batch
$ dotnet construct src
$ cdk bootstrap
$ cdk deploy --require-approval by no means

NOTE: Optionally, it’s also possible to use the “run.sh” script/bash file offered as a part of the code base inside “cdk” folder, that can care for the above steps. As soon as the previous CDK Deploy command is accomplished efficiently, two CloudFormation stacks are created. Take a second to determine the most important parts. The CloudFormation stack spins up the next assets, which will be considered within the AWS Administration Console. Please discover pattern stack assets that will get created. This may be modified with “Constants.cs” file having STACK_PREFIX variable (Ex: netcore is the default prefix within the offered pattern)

  • CloudFormation
    • CDK Toolkit
    • Stack – netcore-cdk-batch-app
  • EC2 (Compute Infrastructure)
    • netcore-batch-compute-environment-asg-<guid>
  • S3 Bucket
    • netcore-batch-processing-job-<account_number>
  • ECR
    • netcore-cdk-batch-app-repository
  • Dynamo Desk
    • netcore-cdk-batch-app-table
  • AWS Batch Job Queue
    • netcore-batch-job-queue Job Definition – netcore-batch-job-definition
  • Lambda Operate
    • netcore-lambda-batch-processing-function

Utilizing CDK constructs, we now have constructed the above infrastructure and built-in the answer with a Public Load Balancer. The output of this stack will give the API URLs for well being examine and API validation. As you discover by defining the answer utilizing CDK, you have been capable of:

  • Use object-oriented strategies to create a mannequin of your system
  • Manage your undertaking into logical modules
  • Save time with code completion in your IDE

Different main benefits utilizing this CDK method embrace, as a developer/improvement group we should always give you the option, to

  • Use logic (if statements, for-loops, and so on) when defining your infrastructure
  • Outline excessive stage abstractions, share them, and publish them to your group, firm, or group
  • Share and reuse your infrastructure as a library
  • Take a look at your infrastructure code utilizing industry-standard protocols
  • Use your present code evaluate workflow


  1. In AWS Console, choose “CloudFormation”. Choose the S3 bucket that was created as a part of the stack.
  2. This shall be one thing like – netcore-batch-processing-job-<account-number>
  3. Drop the pattern CSV file offered as a part of the code.
  4. The processed CSV file rows are saved in Dynamo desk as output

AWS CDK Destroy command deletes the CloudFormation stack and all of the AWS Assets that have been created as a part of the stack. AWS Providers that present “Retain” coverage on stack deletion should be manually deleted. This may be completed both in AWS Console or utilizing AWS CLI (instructions offered) under

Utilizing AWS Console

  1. Open AWS Console, choose “S3”, navigate to the bucket created as a part of the stack and delete the S3 bucket manually.
  2. Equally throughout the AWS Console, search for “ECR” and “DynamoDB” desk which might be created as a part of the stack and delete the repository and desk that have been created by this stack.
  3. In AWS Console, search for “CloudFormation”, choose “CDKToolkit” stack
  4. Go to “Assets” tab, choose the staging s3 bucket
  5. Choose all of the contents & delete the contents manually

Utilizing AWS CLI

$ cdk destroy

# CLI Instructions to delete the S3, Dynamo and ECR repository

$ aws s3 rb s3://netcore-batch-processing-job- --force

$ aws ecr delete-repository --repository-name netcore-cdk-batch-app-repository --force

$ aws dynamodb delete-table --table-name netcore-cdk-batch-app-table

You have been capable of launch an utility course of involving AWS Batch to combine with numerous AWS companies. Relying on the scalability of the appliance wants, AWS Batch is ready to course of each quicker and price effectively. The publish walked via deploying a lambda and batch processing code with infrastructure as code utilizing Microsoft .NET AWS CDK.

I encourage you to check this instance and see for your self how this general orchestration works with AWS Batch. Then, it’s only a matter of changing your Python (or another programming language framework) code, packaging it as a Docker container, and letting the AWS Batch deal with the method effectively. For those who determine to offer it a attempt, have any doubt, or need to let me know what you consider the publish, please go away a remark!

* AWS Cloud Improvement Equipment(CDK)
* AWS CDK .NET API Reference
* Microsoft .Internet Core
* Home windows on AWS
* Docker Containers

In case you are desirous about a .NET Internet API with Amazon Aurora database utilizing AWS CDK, refer this weblog.

Siva Ramani (Sr Cloud App Arch)

Sivasubramanian Ramani (Siva Ramani) is a Sr Cloud Utility Architect at AWS. His experience is in utility optimization, serverless options and utilizing Microsoft utility workloads with AWS.


Source link

Leave a Reply

Your email address will not be published.

Previous Post

3 Ways Your Business is Getting Behind without a Website Presence

Next Post

IBM Launches Quantum Programming Challenge

Related Posts