Skip to content

Commit 164c9a1

Browse files
edwardchchenkaiz-io
edwardchchen
and
kaiz-io
authored
feat(typescript/ddb-stream-lambda-sns) Add an example of DynamoDB Stream integration with Lambda and SNS (#1099)
* feat(typescript/ddb-stream-lambda-sns) Add an example of dynamodb stream integration with lambda and sns * docs(typescript/ddb-stream-lambda-sns) Update README.md * fixes(typescript/ddb-stream-lambda-sns) Remove unused packages --------- Co-authored-by: Michael Kaiser <[email protected]>
1 parent 99ab6b6 commit 164c9a1

File tree

13 files changed

+507
-0
lines changed

13 files changed

+507
-0
lines changed

‎typescript/ddb-stream-lambda-sns/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out
9+
.DS_Store

‎typescript/ddb-stream-lambda-sns/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎typescript/ddb-stream-lambda-sns/README.md‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
# DynamoDB Stream Integration with Lambda and SNS
2+
3+
<!--BEGIN STABILITY BANNER-->
4+
---
5+
6+
![Stability: Developer Preview](https://img.shields.io/badge/stability-Developer--Preview-important.svg?style=for-the-badge)
7+
8+
> **This is an experimental example. It may not build out of the box**
9+
>
10+
> This example is built on Construct Libraries marked "Developer Preview" and may not be updated for latest breaking changes.
11+
>
12+
> It may additionally requires infrastructure prerequisites that must be created before successful build.
13+
>
14+
> If build is unsuccessful, please create an [issue](https://github.com/aws-samples/aws-cdk-examples/issues/new) so that we may debug the problem
15+
---
16+
<!--END STABILITY BANNER-->
17+
18+
## Overview
19+
20+
This repository provides both L2 and L3 constructs example usage for working with DynamoDB streams [AWS Cloud Development Kit (CDK)](https://aws.amazon.com/cdk/) with TypeScript. It showcases the integration of DynamoDB streams with AWS Lambda and Amazon SNS (Simple Notification Service), providing an example of real-time data processing and notification workflows.
21+
22+
This solution demonstrates a use case for real-time notifications: alerting users about low inventory of an item in the system.
23+
24+
## Architecture Diagram
25+
26+
![Architecture Diagram](images/architecture.jpg)
27+
28+
## Features
29+
30+
- L2 (low-level) construct for fine-grained control over DynamoDB streams
31+
- [L3 (high-level)](https://docs.aws.amazon.com/solutions/latest/constructs/aws-dynamodbstreams-lambda.html) construct for simplified, best-practice implementations of DynamoDB streams
32+
- Integration with Lambda functions for stream processing
33+
- Implements an SQS Dead Letter Queue (DLQ) for the Lambda function failure handling
34+
- Shows how to use Amazon SNS to distribute stream processing results or notifications.
35+
36+
37+
## Build, Deploy and Testing
38+
39+
### Prerequisites
40+
41+
Before you begin, ensure you have met the following requirements:
42+
43+
* You have installed the latest version of [Node.js and npm](https://nodejs.org/en/download/)
44+
* You have installed the [AWS CLI](https://aws.amazon.com/cli/) and configured it with your credentials
45+
* You have installed the [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) globally
46+
* You have an AWS account and have set up your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
47+
* You have [bootstrapped your AWS account](https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html) for CDK
48+
49+
50+
51+
### Build
52+
To build this app, you need to be in this example's root folder. Then run the following:
53+
54+
```bash
55+
npm install
56+
npm run build
57+
```
58+
59+
This will install the dependencies for this example.
60+
61+
### Deploy
62+
63+
Run `cdk deploy`. This will deploy the Stack to your AWS Account.
64+
65+
Post deployment, you should see the table arn, lambda function arn and sns topic arn on the output of your CLI.
66+
67+
## Testing
68+
```bash
69+
npm run test
70+
```
71+
72+
## Usage
73+
74+
### Configuring SNS Notification Subscription
75+
76+
1. After deploying the stack, locate the SNS topic Amazon Resource Name (ARN) from the CLI output.
77+
78+
2. To subscribe an email address to the SNS topic:
79+
80+
```bash
81+
aws sns subscribe --topic-arn <your-sns-topic-arn> --protocol email --notification-endpoint [email protected]
82+
```
83+
Replace <your-sns-topic-arn> with the actual ARN of your SNS topic, and [email protected] with the email address you want to subscribe.
84+
85+
3. Check your email inbox for a confirmation message from AWS. Click the link in the email to confirm your subscription.
86+
87+
### Creating an Item in DynamoDB with id, itemName, and count
88+
To trigger the stream processing and email notification, you need to create an item in your DynamoDB table with the fields id, itemName, and count. You can do this using the AWS CLI or AWS Management Console.
89+
90+
Example item.json provided in this repo:
91+
```bash
92+
{
93+
"id": {
94+
"S": "1"
95+
},
96+
"count": {
97+
"N": "10"
98+
},
99+
"itemName": {
100+
"S": "Coffee Beans"
101+
}
102+
}
103+
```
104+
105+
1. Use the following command to put the item into your DynamoDB table:
106+
107+
```bash
108+
aws dynamodb put-item --table-name <your-table-name> --item file://item.json
109+
```
110+
111+
Replace <your-table-name> with the actual name of your DynamoDB table.
112+
113+
2. Whenever you update the count field of the item to 0, the DynamoDB stream will trigger the Lambda function, which will process the data and send a notification to the subscribed email address via SNS.
114+
115+
116+
117+
## Cleanup
118+
119+
To avoid incurring future charges, please destroy the resources when they are no longer needed:
120+
121+
```bash
122+
cdk destroy
123+
```

‎typescript/ddb-stream-lambda-sns/bin/ddb-stream.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/env node
2+
import * as cdk from 'aws-cdk-lib';
3+
import { DdbStreamStack } from '../lib/ddb-stream-stack';
4+
5+
const app = new cdk.App();
6+
new DdbStreamStack(app, 'DdbStreamStack', {});

‎typescript/ddb-stream-lambda-sns/cdk.json‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/ddb-stream.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
38+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
39+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
40+
"@aws-cdk/aws-route53-patters:useCertificate": true,
41+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
42+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
43+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
44+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
45+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
46+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
47+
"@aws-cdk/aws-redshift:columnId": true,
48+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
49+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
50+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
51+
"@aws-cdk/aws-kms:aliasNameRef": true,
52+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
53+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
54+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
55+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
56+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
57+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
58+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
59+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
60+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
61+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
62+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
63+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
64+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
65+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
66+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
67+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
68+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
69+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
70+
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false
71+
}
72+
}

‎typescript/ddb-stream-lambda-sns/images/architecture.jpg‎

70 KB
Loading

‎typescript/ddb-stream-lambda-sns/item.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"id": {
3+
"S": "1"
4+
},
5+
"count": {
6+
"N": "10"
7+
},
8+
"itemName": {
9+
"S": "Coffee Beans"
10+
}
11+
}

‎typescript/ddb-stream-lambda-sns/jest.config.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
module.exports = {
2+
testEnvironment: 'node',
3+
roots: ['<rootDir>/test'],
4+
testMatch: ['**/*.test.ts'],
5+
transform: {
6+
'^.+\\.tsx?$': 'ts-jest'
7+
}
8+
};

‎typescript/ddb-stream-lambda-sns/lib/ddb-stream-stack.ts‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
import * as cdk from 'aws-cdk-lib';
2+
import { Construct } from 'constructs';
3+
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
4+
import * as lambda from 'aws-cdk-lib/aws-lambda';
5+
import * as lambdaEventSources from 'aws-cdk-lib/aws-lambda-event-sources';
6+
import { DynamoDBStreamsToLambda } from '@aws-solutions-constructs/aws-dynamodbstreams-lambda';
7+
import * as sns from 'aws-cdk-lib/aws-sns';
8+
import * as kms from 'aws-cdk-lib/aws-kms';
9+
import * as sqs from 'aws-cdk-lib/aws-sqs';
10+
11+
export class DdbStreamStack extends cdk.Stack {
12+
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
13+
super(scope, id, props);
14+
const aws_sns_kms_key = kms.Alias.fromAliasName(
15+
this,
16+
"aws-managed-sns-kms-key",
17+
"alias/aws/sns",
18+
)
19+
20+
const snsTopic = new sns.Topic(this, 'ddb-stream-topic', {
21+
topicName: 'ddb-stream-topic',
22+
displayName: 'SNS Topic for DDB streams',
23+
enforceSSL: true,
24+
masterKey: aws_sns_kms_key,
25+
});
26+
27+
//L2 CDK Construct
28+
const deadLetterQueueL2 = new sqs.Queue(this, 'ddb-stream-l2-dlq', {
29+
queueName: 'ddb-stream-l2-dlq',
30+
encryption: sqs.QueueEncryption.KMS_MANAGED,
31+
retentionPeriod: cdk.Duration.days(4), // Adjust retention period as needed
32+
});
33+
34+
const itemL2Table = new dynamodb.Table(this, 'itemL2Table', {
35+
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
36+
stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
37+
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
38+
encryption: dynamodb.TableEncryption.AWS_MANAGED,
39+
//If you wish to retain the table after running cdk destroy, comment out the line below
40+
removalPolicy: cdk.RemovalPolicy.DESTROY
41+
});
42+
43+
const itemL2TableLambdaFunction = new lambda.Function(this, 'itemL2TableLambdaFunction', {
44+
runtime: lambda.Runtime.NODEJS_20_X,
45+
handler: 'index.handler',
46+
tracing: lambda.Tracing.ACTIVE,
47+
code: lambda.Code.fromAsset('resources/lambda'),
48+
environment: {
49+
SNS_TOPIC_ARN: snsTopic.topicArn,
50+
AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1'
51+
},
52+
});
53+
itemL2TableLambdaFunction.addEventSource(new lambdaEventSources.DynamoEventSource(itemL2Table, {
54+
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
55+
onFailure: new lambdaEventSources.SqsDlq(deadLetterQueueL2),
56+
bisectBatchOnError: true,
57+
maxRecordAge: cdk.Duration.hours(24),
58+
retryAttempts: 500,
59+
}));
60+
61+
deadLetterQueueL2.grantSendMessages(itemL2TableLambdaFunction);
62+
63+
itemL2Table.grantStreamRead(itemL2TableLambdaFunction);
64+
65+
//L3 CDK Construct
66+
const itemL3Table = new DynamoDBStreamsToLambda(this, 'itemL3Table', {
67+
dynamoTableProps: {
68+
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
69+
stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
70+
//If you wish to retain the table after running cdk destroy, comment out the line below
71+
removalPolicy: cdk.RemovalPolicy.DESTROY
72+
},
73+
lambdaFunctionProps: {
74+
code: lambda.Code.fromAsset('resources/lambda'),
75+
runtime: lambda.Runtime.NODEJS_20_X,
76+
handler: 'index.handler',
77+
environment: {
78+
SNS_TOPIC_ARN: snsTopic.topicArn,
79+
},
80+
},
81+
});
82+
83+
snsTopic.grantPublish(itemL2TableLambdaFunction);
84+
snsTopic.grantPublish(itemL3Table.lambdaFunction);
85+
86+
new cdk.CfnOutput(this, 'itemL2TableLambdaFunctionArn', { value: itemL2TableLambdaFunction.functionArn });
87+
new cdk.CfnOutput(this, 'itemL3TableLambdaFunctionArn', { value: itemL3Table.lambdaFunction.functionArn });
88+
new cdk.CfnOutput(this, 'l3TableArn', { value: itemL3Table.dynamoTableInterface.tableArn });
89+
new cdk.CfnOutput(this, 'l2TableArn', { value: itemL2Table.tableArn });
90+
new cdk.CfnOutput(this, 'topicArn', { value: snsTopic.topicArn });
91+
new cdk.CfnOutput(this, 'l2DLQArn', { value: deadLetterQueueL2.queueArn })
92+
}
93+
}

‎typescript/ddb-stream-lambda-sns/package.json‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"name": "ddb-stream-lambda-sns",
3+
"version": "0.1.0",
4+
"bin": {
5+
"ddb-stream": "bin/ddb-stream.js"
6+
},
7+
"scripts": {
8+
"build": "tsc",
9+
"watch": "tsc -w",
10+
"test": "jest",
11+
"cdk": "cdk"
12+
},
13+
"devDependencies": {
14+
"@types/jest": "^29.5.14",
15+
"@types/node": "22.9.0",
16+
"aws-cdk": "2.166.0",
17+
"jest": "^29.7.0",
18+
"ts-jest": "^29.2.5",
19+
"ts-node": "^10.9.2",
20+
"typescript": "~5.6.3"
21+
},
22+
"dependencies": {
23+
"@aws-solutions-constructs/aws-dynamodbstreams-lambda": "^2.74.0",
24+
"aws-cdk-lib": "2.167.0",
25+
"constructs": "^10.4.2"
26+
}
27+
}

0 commit comments

Comments
 (0)