What is Substation?

Substation is an open source data pipeline and transformation toolkit written in Go that has these features:

• Deploys modular, cloud native data pipelines that support 100s of unique designs
• Offers event-driven ingest, transform, load (ITL) applications that evaluate, process, and deliver data in real-time
• Provides Go packages for creating custom data processing applications

Features

Modular, Cloud Native Data Pipelines and Microservices

• Design unique data pipelines managed using infrastructure as code
  • Modular architecture supports 100s of unique pipeline designs
  • Use microservices to integrate with external systems and users
• Affordable data processing at massive scale
  • Scales beyond 100,000 events per second (EPS)
  • Costs as low as $10/TB for simple pipelines and up to $40/TB for complex pipelines
• Near zero maintenance by running on serverless cloud services

Evaluate, Process, & Deliver Data

• Evaluate, process, and deliver data in real-time
• Support any dataset using configurations as code

Build Custom Data Processing Apps

• Evaluate and filter data using Conditions
• Modify data from, to, and in-place as objects using Processors
• Build custom Substation applications for bespoke ITL use cases

Differentiating Benefits

Substation differentiates itself from other data pipeline solutions in several ways:

• Fully serverless — you’ll never manage a server or think about sizing a cluster
• Designed for scale — automatically scales from 1 to 100,000+ EPS with no intervention required by an engineer
• Infrastructure and configurations as code — uses Terraform, Jsonnet, and AWS AppConfig, which means you can deploy unique, reusable data pipelines in minutes
• Cost efficient — we went the extra mile to make things affordable, including creating a compliant, minimal version of the Kinesis Producer and Client Libraries in Go
• Extensible — we expose the Substation core as Go packages, so you can build your own custom data processing applications