Use Case:-

We have a business use case where we need to call our nightly job for multiple tenants. Initially, I planned to use EventBridge Bus and Rules to achieve this goal. However, we discovered that EventBridge Rules has a limit of 5 targets, which would prevent us from adding more tenants as input for the nightly job. I want to avoid maintaining multiple rules for the same use case, so I came up with this solution. I believe No-Code/Less-Code approach using AWS Step Functions and EventBridge is the best way forward for our development needs.

The Architecture:-

Steps:-

1. Configure EventBridge Schduler. You can read my prior 2 posts here and here.

2. Populate DynamDB static data.

3. Create a Step Function State Machine using AWS ASL language.

You can download the source code from here, which has been implemented using AWS CDK and SAM, whichever approach you prefer.

CDK Deploy:-

1) cd cdk
2) npm install
3) cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>

SAM Deploy:-

1) cd sam
2) sam deploy --stack-name LegacyAppStack --capabilities    CAPABILITY_NAMED_IAM --guided --profile <profile name>

Populate Static Data in DynamoDB:-

1) cd dynamodb
2) ./dynamodb.sh

You will see the below records in the DynamoDB table.

Understand State Machine:-

AWS step functions are integrated with more than 200+ AWS services and you can call 9K+ API from step functions. There are 2 ways you can integrate step functions with other services, either using AWS SDK Integration or you can use Optimized integration. In this example, I am using Default Response (Request/Response) optimized integration pattern with Express workflow.

"GetTenantList": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:getItem",
      "Parameters": {
        "TableName": "LegacyApp",
        "Key": {
          "PK": {
            "S": "METADATA"
          },
          "SK": {
            "S": "METADATA"
          }
        }
      },
      "ResultPath": "$.context",
      "OutputPath": "$.context.Item.TenantList.L",
      "Next": "IterateTenantList"
    }

I am utilizing "arn:aws:states:::dynamodb:getItem" API to retrieve data from DynamoDB by passing the primary key (PK) and sort key (SK). Step functions offer access to all CRUD API operations available with DynamoDB. After filtering output with the ResultPath and OutputPath, the output of the aforementioned operation would be as follows.

[
  {
    "S": "TENANT#123456"
  },
  {
    "S": "TENANT#456789"
  }
]

Now I am iterating TenantList using Map. The Map states concurrently iterate over a collection of items in a dataset, such as a list of S3 Objects, CSV file or JSON object. it repeats a set of steps for each item in the collection.

 "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        }

The Map states have an "Inline" mode where they can only accept input in the form of a JSON array. When operating in this mode, the map states can handle up to 40 concurrent iterations. However, the Map states also offer a "Distributed" mode that can support up to 10,000 parallel child workflows. I am planning to write a blog about these features soon.

Next, I am getting messages from DynamoDB and transforming them before sending them to SNS.

"StartAt": "GetTenant",
        "States": {
          "GetTenant": {
            "Type": "Task",
            "Resource": "arn:aws:states:::dynamodb:getItem",
            "Parameters": {
              "TableName": "LegacyApp",
              "Key": {
                "PK": {
                  "S.$": "$"
                },
                "SK": {
                  "S.$": "$"
                }
              }
            },
            "InputPath": "$.S",
            "Next": "TransformMessage",
            "ResultSelector": {
              "message.$": "States.StringToJson($.Item.Message.S)"
            }
          },
          "TransformMessage": {
            "Type": "Pass",
            "Parameters": {
              "version.$": "$.message.version",
              "source.$": "$.message.source",
              "eventType.$": "$.message.eventType",
              "details": {
                "tenantId.$": "$.message.details.tenantId"
              },
              "correlationId.$": "States.UUID()"
            },
            "Next": "SendMessageToLegacyApp"
          }

Step functions provide several intrinsic functions. You can see here I am using "States.StringToJson" and "States.UUID". Intrinsic functions will help you to perform basic data processing operations without using a Task state. Here is the output of the above 2 tasks.

{
  "details": {
    "tenantId": 123456
  },
  "correlationId": "f43e0b5b-c187-483f-800e-44fb71486494",
  "source": "SampleApp",
  "eventType": "LegacyApp",
  "version": "1.0"
}

In the last step, I am sending messages to the SNS topic using the following code.

 "SendMessageToLegacyApp": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sns:publish",
            "Parameters": {
              "Message.$": "$",
              "TopicArn": "${LegacyAppSNSPath}"
            },
            "End": true
          }

Cleanup:-

sam delete --stack-name LegacyAppStack --profile <your profile name>
                               OR
cdk destroy --profile <your profile name>

Wrap-Up:-

We have not utilized any Lambda code in our implementation, relying entirely on AWS API integration. I believe that each line of code in any service is a liability, it carries a certain amount of risk, and it requires ongoing maintenance, monitoring, and the creation of unit/integration test cases. Let me know if you have any better way to solve this use case.

Code is more art to me. I don’t want computer art - Ben Pyle