HTTP

HTTP processes values by retrieving data from or sending data to an HTTP(S) URL. Multiple use cases are supported, including:

• Enrich data with any HTTP(S) service
• Get data from any HTTP(S) endpoint
• Send objects to any HTTP(S) endpoint

This processor uses an exponential retry strategy that can take up to 2 minutes for each HTTP request and may have significant impact on end-to-end data processing latency. If Substation is running in AWS Lambda with Kinesis, then this latency can be mitigated by increasing the parallelization factor of the Lambda or by implementing a caching strategy.

Interpretation Methods

The processor supports these interpretation methods:

• object
• data

Data Interpolation

Data can be optionally interpolated with the URL by placing the string ${data} anywhere in the URL. For example:

Configured URL	Data	Interpolated URL
hxxps://foo.com/path/to/${data}	{"addr":"8.8.8.8"}	hxxps://foo.com/path/to/8.8.8.8
hxxps://foo.com/path/to/${data}	8.8.8.8	hxxps://foo.com/path/to/8.8.8.8

Secrets Interpolation

Secrets can be optionally interpolated with the URL and header values. Multiple secrets can be interpolated in a single string. For example:

Configured URL	Data	Environment Variables	Interpolated URL
hxxps://foo.com/path/to/${data}?token=${SECRETS_ENV:TOKEN}	{"addr":"8.8.8.8"}	TOKEN=mysecret	hxxps://foo.com/path/to/8.8.8.8?token=mysecret
hxxps://foo.com/path/to/${data}?token=${SECRETS_ENV:TOKEN}&user=${SECRETS_ENV:USERNAME}	{"addr":"8.8.8.8"}	TOKEN=mysecret USERNAME=myusername	hxxps://foo.com/path/to/8.8.8.8?token=mysecret&user=myusername

Options

Field	Type	Description	Required
url	string	the HTTP(S) URL used in the request. URLs support loading secrets.	Yes
method	string	the HTTP method used in the request. must be one of: - GET - POST defaults to GET.	No
headers	[]object	an array of objects that contain HTTP headers sent in the request. header values support loading secrets. defaults to no headers.	No
body_key	string	retrieves a value from an object that is used as the message body. this is only used in HTTP requests that send payloads to the server (e.g., POST). defaults to no body.	No

Use Cases

Enriching Data with GreyNoise

Substation ships with a pattern for enriching data with GreyNoise, including bulk enrichment. Use this recipe to get started:

⚙️

Enriching Data with GreyNoise

Open Recipe

GreyNoise rate limits apply. For any questions, please reach out to [email protected].

Downloading Text Files

The processor can be used to download text files from any HTTP(S) endpoint. For example, if data interpolation isn't used then it can download Moby Dick by Herman Melville:

URL	Capsule
https://www.gutenberg.org/files/2701/old/moby10b.txt	{"moby_dick":"**The Project Gutenberg Etext of Moby Dick, by Herman Melville**"}

This can be combined with the Copy processor's data from object pattern to overwrite the original data, leaving only the downloaded file.

Downloading Non-Text Files

The processor can also be used to retrieve non-text (e.g., binary) files. If the processor is configured to interpret data (omitting key and set_key), then the file overwrites the capsule's data with no modification, but if the processor is configured to interpret objects, then the file is base64 encoded before it is written to the object. For example, it can be used to download a PDF version of Moby Dick:

Interpretation Method	URL	Capsule
Data	http://www.gasl.org/refbib/Melville__Moby_Dick.pdf	%PDF-1.5
Object	http://www.gasl.org/refbib/Melville__Moby_Dick.pdf	{"moby_dick":"JVBERi0xLjU="}