---
agentTools:
  projectIndex: https://substation.readme.io/llms.txt
---

# Utility

Utility transforms provide testing or debugging functions.

# utility.control

Generates a control [message](https://substation.readme.io/docs/messages) based on a [batch](https://substation.readme.io/docs/batching-data) configuration.

## Settings

[block:parameters]
{
  "data": {
    "h-0": "Field",
    "h-1": "Type",
    "h-2": "Description",
    "h-3": "Required",
    "0-0": "batch.count",
    "0-1": "int",
    "0-2": "Maximum number of items to buffer before emitting a new array.  \n  \nDefaults to 1,000 items.",
    "0-3": "No",
    "1-0": "batch.size",
    "1-1": "int",
    "1-2": "Maximum size (in bytes) of items to buffer before emitting a new array.  \n  \nDefaults to 1MB.",
    "1-3": "No",
    "2-0": "batch.duration",
    "2-1": "string",
    "2-2": "Maximum duration to buffer items before emitting a new array.  \n  \nDefaults to 1m.",
    "2-3": "No"
  },
  "cols": 4,
  "rows": 3,
  "align": [
    "left",
    "left",
    "left",
    "left"
  ]
}
[/block]

## Example

```Text Config
sub.transform.utility.control(
  settings={ batch: { count: 1000 } }
)
```
```Text Abbv. Config
sub.tf.util.ctrl({ batch: { count: 1000 } })
```

# utility.delay

Delays transformation for a duration of time.

## Settings

| Field    | Type   | Description                | Required |
| :------- | :----- | :------------------------- | :------- |
| duration | string | Duration of time to sleep. | Yes      |

## Example

```Text Config
sub.transform.utility.delay(
  settings={duration: '1m'}
)
```
```Text Abbv. Config
sub.tf.util.delay({duration: '1m'})
```

# utility.drop

Drops the message and returns nothing.

## Example

```Text Config
sub.transform.utility.drop()
```
```Text Abbv. Config
sub.tf.util.drop()
```

# utility.err

Generates an error with a custom message.

## Settings

| Field   | Type   | Description                    | Required |
| :------ | :----- | :----------------------------- | :------- |
| message | string | Message returned in the error. | Yes      |

## Example

```Text Config
sub.transform.utility.err(
  settings={message: 'test'}
)
```
```Text Abbv. Config
sub.tf.util.err({message: 'test'})
```

# utility.metric.bytes

Generates a [metric](https://substation.readme.io/docs/metrics-destinations) that reports the sum of [message](https://substation.readme.io/docs/messages) bytes that have passed through the transform.

## Settings

| Field              | Type   | Description                                                                                                  | Required |
| :----------------- | :----- | :----------------------------------------------------------------------------------------------------------- | :------- |
| metric.name        | string | Name of the metric.                                                                                          | Yes      |
| metric.destination | object | [Metrics Destination](https://substation.readme.io/docs/metrics-destinations) configuration that reports the metric to an external system. | Yes      |
| metric.attributes  | map    | Map (dictionary) of strings that are included in the metric as attributes or labels.                         | No       |

## Example

```Text Config
sub.transform.utility.metric.bytes(
  settings={ metric: { name: 'BytesReceived', destination: { type: 'aws_cloudwatch_embedded_metrics' } } },
)
```
```Text Abbv. Config
sub.tf.util.metric.bytes({ metric: { name: 'BytesReceived', destination: { type: 'aws_cloudwatch_embedded_metrics' } } })
```

# utility.metric.count

Generates a [metric](https://substation.readme.io/docs/metrics-destinations) that reports the count of [messages](https://substation.readme.io/docs/messages) that have passed through the transform.

## Settings

| Field              | Type   | Description                                                                                                  | Required |
| :----------------- | :----- | :----------------------------------------------------------------------------------------------------------- | :------- |
| metric.name        | string | Name of the metric.                                                                                          | Yes      |
| metric.destination | object | [Metrics Destination](https://substation.readme.io/docs/metrics-destinations) configuration that reports the metric to an external system. | Yes      |
| metric.attributes  | map    | Map (dictionary) of strings that are included in the metric as attributes or labels.                         | No       |

## Example

```Text Config
sub.transform.utility.metric.count(
  settings={ metric: { name: 'MessagesReceived', destination: { type: 'aws_cloudwatch_embedded_metrics' } } },
)
```
```Text Abbv. Config
sub.tf.util.metric.count({ metric: { name: 'MessagesReceived', destination: { type: 'aws_cloudwatch_embedded_metrics' } } })
```

# utility.metric.freshness

Generates [metrics](https://substation.readme.io/docs/metrics-destinations) that reports the [freshness](https://sre.google/workbook/data-processing/) of [messages](https://substation.readme.io/docs/messages) that have passed through the transform.

Freshness is measured by the difference between a time field in the message and the time the message passes through the transform. This generates "success" and "failure" metrics (i.e. fresh and stale) that are distinguishable using the attribute named `FreshnessType`.

This can be used to measure the freshness of individual pipeline components, segments of a pipeline, or an end-to-end pipeline.

## Settings

| Field              | Type   | Description                                                                                                  | Required |
| :----------------- | :----- | :----------------------------------------------------------------------------------------------------------- | :------- |
| threshold          | string | Delta between the two datetime values that determines if a message is fresh or stale.                        | Yes      |
| object.source\_key |        | Retrieves a value from an object for measurement.                                                            | Yes      |
| metric.name        | string | Name of the metric.                                                                                          | Yes      |
| metric.destination | object | [Metrics Destination](https://substation.readme.io/docs/metrics-destinations) configuration that reports the metric to an external system. | Yes      |
| metric.attributes  | map    | Map (dictionary) of strings that are included in the metric as attributes or labels.                         | No       |

## Example

```Text Config
sub.transform.utility.metric.freshness(
  settings={ 
     threshold: '5m',
     object: { source_key: 'timestamp' } },
     metric: { name: 'MessageFreshness', destination: { type: 'aws_cloudwatch_embedded_metrics' } } 
  },
)
```
```Text Abbv. Config
sub.tf.util.metric.freshness({
  threshold: '5m',
  obj: { src: 'timestamp' } },
  metric: { name: 'MessagesReceived', destination: { type: 'aws_cloudwatch_embedded_metrics' } }
})
```

<br />

# utility.secret

Retrieves a secret from an external secret management system and loads it into the internal [Secrets Store](https://substation.readme.io/docs/secrets-store).

## Settings

| Field  | Type   | Description                                                     | Required |
| :----- | :----- | :-------------------------------------------------------------- | :------- |
| secret | object | The Secret configuration for retrieving and loading the secret. | Yes      |

## Example

```Text Config
sub.transform.utility.secret(
  settings={ secret: sub.secrets.environment_variable(
    settings={ id: 'ENV_VAR', name: 'SUBSTATION_SECRET' }
  ) }
)
```
```Text Abbv. Config
sub.tf.util.secret({ secret: sub.secrets.environment_variable({ id: 'ENV_VAR', name: 'SUBSTATION_SECRET' }) })
```

# Use Cases

## Conditionally Drop Messages

Messages can be conditionally dropped by wrapping the `utility.drop`  transform with `meta.switch`:

```Text config.jsonnet
sub.transform.meta.switch(
  settings={ cases: [
    condition=[...],
    transform=sub.tf.utility.drop(),
  ]},
)
```