Meta
Meta transforms apply other transforms.
meta.err
Catches errors returned by other transforms and returns messages.
Settings
Field | Type | Description | Required |
---|---|---|---|
transforms | []object | Transforms that are applied in series. | Yes |
error_messages | []array | Regular expressions that match error messages and determine if the error should be caught. | Yes |
Example
sub.transform.meta.err({transforms: [
sub.transform.enrich.http.get([...])}
]})
sub.tf.meta.err({transforms: [sub.tf.enrich.http.get([...])]})
meta.for_each
Applies transform to every element of a JSON array.
Settings
Field | Type | Description | Required |
---|---|---|---|
object.source_key | string | Retrieves a value from an object for transformation. | Yes |
object.target_key | string | Places a value into an object after transformation. | Yes |
transforms | []object | Transforms that are applied in series. | Yes |
Example
sub.transform.meta.for_each({
object: {source_key: 'arr', target_key: 'arr'},
transforms: [
sub.transform.enrich.http.get([...]),
],
})
sub.tf.meta.for_each({
object: {source_key: 'arr', target_key: 'arr'},
transforms: [
sub.tf.enrich.http.get([...]),
],
})
meta.kv_store.lock
Transforms data by acquiring a lock in a key-value store before executing a transform. Any error in the transform will cause keys to unlock.
This transform can be used to achieve "exactly-once" semantics in other transforms, including send transforms.
Settings
Field | Type | Description | Required |
---|---|---|---|
object.source_key | string | Retrieves a value from an object that is used as the key to lock the item in the KV store. Defaults to an empty string (the SHA256 hash of the message is the key). | No |
object.ttl_key | string | Retrieves a value from an object that is used as the time-to-live (TTL) of the item set into the KV store. This value must be an integer that represents the Unix time when the item will be evicted from the store. Any precision greater than seconds (e.g., milliseconds, nanoseconds) is truncated to seconds. Defaults to an empty string (no TTL is used when locking items in the store). | No |
prefix | string | String that is prepended to the value retrieved by object.source_key .Defaults to an empty string (no prefix is used). | No |
ttl_offset | string | An offset used to determine the time-to-live (TTL) of the item set into the KV store. If object.ttl_key is configured, then this value is added to the TTL value retrieved from the object. If object.ttl_key is not used, then this value is added to the current time.For example, if object.ttl_key is not configured and the offset is "1d" (1 day), then the value will be evicted from the store when more than 1 day from now has passed.Defaults to an empty string (no TTL is used when locking items in the store). | No |
transforms | []object | Transforms that are applied in series. | Yes |
kv_store | object | The KV store configuration settings. Refer to each KV store backend described in Key-Value Stores for more information. | Yes |
Example
sub.transform.meta.kv_store.lock({
kv_store: kv,
ttl_offset: '1m',
transforms: [
sub.transform.send.stdout(), // Only unique messages are sent to stdout.
],
})
sub.tf.meta.kv.lock(settings={
kv_store: kv,
ttl_offset: '1m',
transforms: [
sub.tf.send.stdout(), // Only unique messages are sent to stdout.
],
})
meta.metric.duration
Generates a metric that reports the execution time (duration) of the transform.
Settings
Field | Type | Description | Required |
---|---|---|---|
metric.name | string | Name of the metric. | Yes |
metric.destination | object | Metrics Destination configuration that reports the metric to an external system. | Yes |
transforms | []object | Transforms that are applied in series. | Yes |
metric.attributes | map | Map (dictionary) of strings that are included in the metric as attributes or labels. | No |
Example
sub.transform.meta.metric.duration({
metric: { name: 'ObjectCopyDuration', destination: { type: 'aws_cloudwatch_embedded_metrics' } },
transforms: [
sub.transform.object.copy({ object: { source_key: 'a', target_key: 'z' } }),
],
})
sub.tf.meta.metric.duration({
metric: { name: 'ObjectCopyDuration', destination: { type: 'aws_cloudwatch_embedded_metrics' } },
transforms: [
sub.tf.object.copy({ object: { source_key: 'a', target_key: 'z' } }),
],
})
meta.switch
Conditionally applies transforms using conditional statements:
- if
- if ... else
- if ... elif ... else
Settings
Field | Type | Description | Required |
---|---|---|---|
cases.condition | []object | Array of conditions that apply a transform. | No |
cases.transforms | []object | Array of transforms conditionally applied. | Yes |
Example
sub.transform.meta.switch(
settings={ cases: [
{condition: [...], transforms: [...]},
{condition: [...], transforms: [...]},
]}
)
Use Cases
If Statement
An if statement is created by configuring a single meta.switch
statement. This examples copies a string value if it contains a specific character:
sub.tf.meta.switch(
settings={ cases: [
{
condition: ssub.cnd.string.contains(object: {source_key: 'a'}, value: 'b'}),
transforms:[
sub.tf.object.copy(settings={object: {source_key: 'a', target_key: 'z'}}),
]
}
]}
)
sub.tf.meta.switch(
settings={ switch: [
{
condition: sub.cnd.str.has(settings={object: {src: 'a'}, value: 'b'}),
transforms: [
sub.tf.obj.cp(settings={object: {src: 'a', trg: 'z'}}),
],
}
]}
)
If ... Else Statement
An if ... else statement is created by configuring two meta.switch
statements:
- The first statement is configured with a condition
- The second statement is configured without a condition
This example aggregates JSON text into an array, otherwise the messages are dropped if they are not valid JSON.
sub.tf.meta.switch(
settings={ cases: [
{
condition: sub.cnd.format.json(),
transforms: [
sub.tf.aggregate.to.array(),
],
},
{
transforms: [
sub.tf.utility.drop(),
],
}
]}
)
sub.tf.meta.switch(
settings={ switch: [
{
condition: sub.cnd.fmt.json(),
transforms: [
sub.tf.agg.to.arr(),
],
},
{
transforms: [
sub.tf.util.drop(),
],
}
]}
)
More statements can be added to create if ... elif ... else statements.
Updated 3 months ago