What is Substation?
Substation is an open source data pipeline and transformation toolkit written in Go that has these features:
- Deploys modular, cloud native data pipelines that support 100s of unique designs
- Offers event-driven ingest, transform, load (ITL) applications that evaluate, process, and deliver data in real-time
- Provides Go packages for creating custom data processing applications
Features
Modular, Cloud Native Data Pipelines and Microservices
- Design unique data pipelines managed using infrastructure as code
- Modular architecture supports 100s of unique pipeline designs
- Use microservices to integrate with external systems and users
- Affordable data processing at massive scale
- Scales beyond 100,000 events per second (EPS)
- Costs as low as $10/TB for simple pipelines and up to $40/TB for complex pipelines
- Near zero maintenance by running on serverless cloud services
Evaluate, Process, & Deliver Data
- Evaluate, process, and deliver data in real-time
- Support any dataset using configurations as code
Build Custom Data Processing Apps
- Evaluate and filter data using Conditions
- Modify data from, to, and in-place as objects using Processors
- Build custom Substation applications for bespoke ITL use cases
Differentiating Benefits
Substation differentiates itself from other data pipeline solutions in several ways:
- Fully serverless — you’ll never manage a server or think about sizing a cluster
- Designed for scale — automatically scales from 1 to 100,000+ EPS with no intervention required by an engineer
- Infrastructure and configurations as code — uses Terraform, Jsonnet, and AWS AppConfig, which means you can deploy unique, reusable data pipelines in minutes
- Cost efficient — we went the extra mile to make things affordable, including creating a compliant, minimal version of the Kinesis Producer and Client Libraries in Go
- Extensible — we expose the Substation core as Go packages, so you can build your own custom data processing applications
Updated over 1 year ago