Aggregate

Aggregate processes values buffering and aggregating them into a single item.

Multiple aggregation patterns are supported, including:

• aggregate values using a separator string
• aggregate values into an object array
• aggregate nested objects into object arrays based on unique keys

Interpretation Methods

The processor supports these interpretation methods:

• object
• data

Options

Field	Type	Description	Required
max_count	integer	the maximum number of items stored in the buffer before emitting aggregated items. defaults to 1000 items.	No
max_size	integer	the maximum size (in bytes) of items stored in the buffer before emitting aggregated items. defaults to 10,000 (10KB).	No
separator	string	string that joins the aggregated values. only used when aggregating values as data.	No
key	string	retrieves a value from an object that is used to sort aggregated items. only used when aggregating values in objects.	No

📘

Aggregated items are emitted when either max_count or max_size is reached!

Use Cases

Aggregating Values as Data

The processor uses these options for aggregating values as data:

• separator: string that joins the aggregated values

For example, if the data aggregation pattern is applied and the separator is \n then the result is:

Input

a 
b 
c 
d

Output

a\nb\nc\d

Input

{"a":"b"} 
{"c":"d"}

Output

{"a":"b"}\n{"c":"d"}

Aggregating Values in an Object Array

The processor uses these options for aggregating values in an object array:

• set_key: where aggregated values are inserted into the new object

For example, if the object array aggregation pattern is applied and the set_key is x.-1, then the result is:

Input

a 
b 
c 
d

Output

{"x":["a","b","c","d"]}

Input

{"a":"b"} 
{"c":"d"}

Output

{"x":[{"a":"b"},{"c":"d"}]}

Aggregating Values into Sorted Object Arrays

The processor uses these options for aggregating values into sorted object arrays:

• set_key: where aggregated values are inserted into the new object
• key: value from the object that is used to sort aggregated items

For example, if the object array aggregation pattern is applied, the set_key is x.-1, and the processor options key is a, then the result is:

Input

{"a":"b"} 
{"a":"c"}
{"a":"b"} 
{"a":"d"} 
{"a":"b"}

Output

{"x":[{"a":"b"},{"a":"b"},{"a":"b"}]}
{"x":[{"a":"c"}]}
{"x":[{"a":"d"}]}

This pattern can be used to reduce data volume by summarizing like objects into an individual object. For example, the first output line above can be chained with the Copy processor to create the object {"a":"b","count":3}.

See this recipe for a detailed walkthrough:

⚙️

Reduce Events through Aggregation

Open Recipe