The project comes with several Terraform modules for deploying resources to build unique data pipelines. Use this recipe to deploy a pipeline that uses nearly every feature of the system:

🚀

Deploying AWS Pipelines

Open Recipe

Using Terraform

An overview on how Terraform is used and the write, plan, apply workflow is available here.

Use the official Terraform documentation for further details on using Terraform for deployment.

AWS

API Gateway

This module deploys API Gateway endpoints.

Kinesis

Creates an API Gateway that puts a record into a Kinesis Data Stream. To prevent hot shards, the partition key is the request ID of the HTTP request.

Lambda

Creates an API Gateway that invokes and sends the HTTP request to a Lambda function.

DynamoDB

This module creates DynamoDB tables that have autoscaling enabled. These tables have time to live enabled and users can optionally use it by writing values to the TTL column.

ECR

This module deploys image repositories to the Elastic Container Registry (ECR) service.

There are two things to be aware of when deploying new image repositories:

Terraform does not manage the build and deployment of container images
- After an image repository is created, then container build and upload should happen externally via Docker
By default all Lambda deployments use containers, but a Lambda deployment will fail if the image repository is empty
- Prevent deployment failures by building and uploading images before deploying Lambda

Event Bridge

Lambda

This module creates Event Bridge rules that trigger a Lambda.

IAM

This module provides default Identity and Access Management (IAM) policies for the most commonly used permissions.

Substation uses this naming convention: [AWS service]_[read|write|modify]_policy. For example, the kinesis_read_policy grants all permissions required to read data from a Kinesis Data Stream.

IAM Attachment

This module attaches policies from the IAM module to resources used in a pipeline (such as Kinesis Data Streams, KMS keys, DynamoDB tables, etc.).

Separating policies and policy attachment allows for granular permission control. We recommend granting least privilege whenever possible.

Kinesis

This module creates new Kinesis Data Streams (KDS) and accompanying CloudWatch alarms.

The streams created by this module are intended to be used with Substation's autoscaling application -- this app provides stream autoscaling at a significantly reduced cost compared to Kinesis Firehose and Kinesis Data Streams On-Demand.

KMS

This module creates encryption keys using the Key Management Service. Any number of keys can be made, but by default we create one key that is shared by all encrypted resources and data.

Lambda

This module creates Lambda that execute Substation applications. When new Lambda are created an accompanying AppConfig configuration profile is created under the substation application.

S3

This module deploys S3 buckets. These buckets are private and objects are encrypted.

WORM

Creates a write once, read many (WORM) S3 bucket using Object Lock and applies the Compliance retention mode setting. Objects written to these buckets cannot be deleted.

SNS

This module creates SNS topics. Topics can be standard or first-in-first-out (FIFO).

SQS

This module creates SQS queues. Queues can be standard or first-in-first-out (FIFO).