Substation is an open-source cloud-native, event-driven data pipeline toolkit designed for security and observability teams that provides three unique capabilities:

Deploy modular, serverless data pipelines in minutes
Evaluate and transform event logs in real-time
Create custom data processing applications written in Go

Features

Deploy Modular, Cloud Native Data Pipelines

Design data pipelines based on your unique use cases and requirements
Autoscale beyond 100,000 events per second with almost zero maintenance
Route data to SIEMs, data lakes, and other log management platforms

Evaluate & Transform Data

Evaluate data before applying transformation functions and routing decisions
Normalize data to a common schema for easy analysis and correlation
Enrich data with threat, infrastructure, and business context

Build Custom Data Processing Apps

Build Substation applications that run in any cloud environment or on-prem
Use Substation's Go packages to evaluate and transform data in your own applications

Benefits

Substation differentiates itself from other data pipeline solutions in several ways:

Fully serverless: You’ll never manage a server or think about sizing a cluster.
Designed for scale: Automatically scales from 1 to 100,000+ EPS with no intervention required by an engineer.
Infrastructure and configurations as code: Uses Terraform, Jsonnet, and AWS AppConfig, which means you can deploy unique, reusable data pipelines in minutes.
Extensible: Substation is open-source and includes Go packages, just in case you want to build your own data processing applications.

Architecture

Substation is built on:

Go (applications, conditions, transforms)
AWS (serverless cloud services)
Jsonnet (configurations as code)
Terraform (infrastructure as code)

Design

Substation supports all of the deployment patterns shown below, and many more:

Packages

Substation applications are built from three Go packages:

Messages (message)
Conditions (condition)
Transforms (transform)

These packages are public and can be used to build custom Substation applications or add data processing to any Go project.