AWS Kinesis sinks KPL-compliant aggregated records to the AWS Kinesis Data Streams service.

This sink can automatically redistribute data across shards by retrieving partition keys from objects; by default, it uses random strings to avoid hot shards.

The sink enforces these ingest restrictions for the Kinesis Data Streams service:

aggregated records cannot be more than 1 MB in size
aggregated records cannot contain more than 10000 records

Interpretation Methods

The sink supports these interpretation methods:

object
data

Settings

Field	Type	Description	Required
stream	string	the Kinesis Data Stream that records are sent to.	Yes
partition	string	string that is used as the partition key for each aggregated record.	No
partition_key	string	retrieves a value from an object that sorts records and is used as the partition key for each aggregated record. if used, then this overrides partition.	No
shard_redistribution	bool	determines if records should be redistributed across shards based on the partition key. defaults to false (data is randomly distributed across shards). if enabled with an empty partition key, then data aggregation is disabled.	No

Use Cases

Redistributing Data Across Shards

If both partition_key and shard_redistribution options are used, then the sink sorts and redistributes data across shards. Redistributing data guarantees delivery of specific data (identified by the partition_key value) to the same Kinesis consumer at the expense of potentially creating hot shards. Read about sharding strategies and hot shards here.

By default, the sink will randomly distribute data to avoid hot shards.