HomeGuides
Try a DemoGet the Latest ReleaseSee the CHANGELOGCreate a Feature Request / Bug ReportJoin the Conversation
Guides

Meta

Meta transforms apply other transforms.

meta.err

Catches errors returned by other transforms and returns messages.

Settings

FieldTypeDescriptionRequired
transforms[]objectTransforms that are applied in series.Yes
error_messages[]arrayRegular expressions that match error messages and determine if the error should be caught.Yes

Example

sub.transform.meta.err({transforms: [
  sub.transform.enrich.http.get([...])}
]})
sub.tf.meta.err({transforms: [sub.tf.enrich.http.get([...])]})

meta.for_each

Applies transform to every element of a JSON array.

Settings

FieldTypeDescriptionRequired
object.source_keystringRetrieves a value from an object for transformation.Yes
object.target_keystringPlaces a value into an object after transformation.Yes
transforms[]objectTransforms that are applied in series.Yes

Example

sub.transform.meta.for_each({
  object: {source_key: 'arr', target_key: 'arr'}, 
  transforms: [
    sub.transform.enrich.http.get([...]),
  ],
})
sub.tf.meta.for_each({
  object: {source_key: 'arr', target_key: 'arr'}, 
  transforms: [
    sub.tf.enrich.http.get([...]),
  ],
})

meta.kv_store.lock

Transforms data by acquiring a lock in a key-value store before executing a transform. Any error in the transform will cause keys to unlock.

This transform can be used to achieve "exactly-once" semantics in other transforms, including send transforms.

Settings

FieldTypeDescriptionRequired
object.source_keystringRetrieves a value from an object that is used as the key to lock the item in the KV store.

Defaults to an empty string (the SHA256 hash of the message is the key).
No
object.ttl_keystringRetrieves a value from an object that is used as the time-to-live (TTL) of the item set into the KV store. This value must be an integer that represents the Unix time when the item will be evicted from the store. Any precision greater than seconds (e.g., milliseconds, nanoseconds) is truncated to seconds.

Defaults to an empty string (no TTL is used when locking items in the store).
No
prefixstringString that is prepended to the value retrieved by object.source_key.

Defaults to an empty string (no prefix is used).
No
ttl_offsetstringAn offset used to determine the time-to-live (TTL) of the item set into the KV store. If object.ttl_key is configured, then this value is added to the TTL value retrieved from the object. If object.ttl_key is not used, then this value is added to the current time.

For example, if object.ttl_key is not configured and the offset is "1d" (1 day), then the value will be evicted from the store when more than 1 day from now has passed.

Defaults to an empty string (no TTL is used when locking items in the store).
No
transforms[]objectTransforms that are applied in series.Yes
kv_storeobjectThe KV store configuration settings. Refer to each KV store backend described in Key-Value Stores for more information.Yes

Example

sub.transform.meta.kv_store.lock({
  kv_store: kv,
  ttl_offset: '1m',
  transforms: [
    sub.transform.send.stdout(),  // Only unique messages are sent to stdout.
  ],
})
sub.tf.meta.kv.lock(settings={
  kv_store: kv,
  ttl_offset: '1m',
  transforms: [
    sub.tf.send.stdout(),  // Only unique messages are sent to stdout.
  ],
})

meta.metric.duration

Generates a metric that reports the execution time (duration) of the transform.

Settings

FieldTypeDescriptionRequired
metric.namestringName of the metric.Yes
metric.destinationobjectMetrics Destination configuration that reports the metric to an external system.Yes
transforms[]objectTransforms that are applied in series.Yes
metric.attributesmapMap (dictionary) of strings that are included in the metric as attributes or labels.No

Example

sub.transform.meta.metric.duration({
  metric: { name: 'ObjectCopyDuration', destination: { type: 'aws_cloudwatch_embedded_metrics' } },
  transforms: [
    sub.transform.object.copy({ object: { source_key: 'a', target_key: 'z' } }),
  ],
})
sub.tf.meta.metric.duration({
  metric: { name: 'ObjectCopyDuration', destination: { type: 'aws_cloudwatch_embedded_metrics' } },
  transforms: [
    sub.tf.object.copy({ object: { source_key: 'a', target_key: 'z' } }),
  ],
})

meta.switch

Conditionally applies transforms using conditional statements:

  • if
  • if ... else
  • if ... elif ... else

Settings

FieldTypeDescriptionRequired
cases.condition[]objectArray of conditions that apply a transform.No
cases.transforms[]objectArray of transforms conditionally applied.Yes

Example

sub.transform.meta.switch(
  settings={ cases: [
    {condition: [...], transforms: [...]},
    {condition: [...], transforms: [...]},
  ]}
)

Use Cases

If Statement

An if statement is created by configuring a single meta.switch statement. This examples copies a string value if it contains a specific character:

sub.tf.meta.switch(
  settings={ cases: [
    { 
      condition: ssub.cnd.string.contains(object: {source_key: 'a'}, value: 'b'}),
      transforms:[
        sub.tf.object.copy(settings={object: {source_key: 'a', target_key: 'z'}}),
      ]
    }
  ]}
)
sub.tf.meta.switch(
  settings={ switch: [
    { 
      condition: sub.cnd.str.has(settings={object: {src: 'a'}, value: 'b'}),
      transforms: [
        sub.tf.obj.cp(settings={object: {src: 'a', trg: 'z'}}),
      ],
    }
  ]}
)

If ... Else Statement

An if ... else statement is created by configuring two meta.switch statements:

  • The first statement is configured with a condition
  • The second statement is configured without a condition

This example aggregates JSON text into an array, otherwise the messages are dropped if they are not valid JSON.

sub.tf.meta.switch(
  settings={ cases: [
    { 
      condition: sub.cnd.format.json(),
      transforms: [
        sub.tf.aggregate.to.array(),
      ],
    },
    {
      transforms: [
        sub.tf.utility.drop(),
      ],
    }
  ]}
)
sub.tf.meta.switch(
  settings={ switch: [
    { 
      condition: sub.cnd.fmt.json(),
      transforms: [
        sub.tf.agg.to.arr(),
      ],
    },
    {
      transforms: [
        sub.tf.util.drop(),
      ],
    }
  ]}
)

More statements can be added to create if ... elif ... else statements.