Simpler Observability with Lambda Telemetry API

AWS constantly improves the overall performance and tooling around AWS Lambda. The recent addition to this trend is the Lambda Telemetry API which enables its users to have increased observability around their Lambda functions.

You can now create your own extensions that use the Telemetry API to subscribe to enhanced telemetry streams directly from Lambda. The extension can then further process them or send them to a place of your choosing. In more detail, there are three different telemetry streams for extensions to subscribe to. These are; Function logs, Extension logs, and Platform telemetry data which have metrics and traces that describe events in the execution environment.

As Thundra, we wanted to make a small contribution to this new feature and open-source a Lambda layer that exports this telemetry information to an S3 bucket of the user’s choosing. You can find the layer and further information in this repository: https://github.com/thundra-io/lambda-telemetry-api-exporter

Lambda Execution Environment

Before the Telemetry API was introduced, the Lambda execution environment looked like this:

You can learn more about the Lambda Extension API here on AWS documentation. With Telemetry API, the Lambda execution environment now looks like this. Extensions can use Telemetry API to receive telemetry streams from the Lambda platform.

You can learn more about the Telemetry API and its extension implementation from the AWS blog post.

Using Thundra Telemetry API Exporter

We decided to write a simple extension that exports the collected telemetry data to an Amazon Kinesis Data Firehose delivery stream. Once the data is delivered to the Firehose stream, the destination is up to the users. You can deliver it directly to an S3 bucket, to a database, or you can handle it by yourself.

The extension is still in development and there is certainly room for improvement. However, it’s still a good proof-of-concept work that shows what you can do with the new Telemetry API. The extension layer currently supports NodeJS runtimes and is available in almost all regions.

To learn more about the exporter, you can visit the public GitHub repository. As to show how it works, the requirements are simple. First, you need to have your Firehose Delivery Stream ready. If you don’t know what Firehose is, you can visit the AWS Documentation page here.

Once that’s ready you can add the below layer to your lambda by replacing the {region} with yours:

arn:aws:lambda:{region}:269863060030:layer:lambda-telemetry-api-exporter:1

The next step would be adding permissions and setting the environment variable that’s required by the Layer. Your Lambda function should have access to the Firehose stream, so a simple permission policy would look like the following. Make sure to replace the delivery stream ARN with the correct one.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "firehose:DeleteDeliveryStream",
                "firehose:PutRecord",
                "firehose:PutRecordBatch",
                "firehose:UpdateDestination"
            ],
            "Resource": [
                // Replace this with yours.
                "arn:aws:firehose:region:account-id:deliverystream/delivery-stream-name"
            ]
        }
    ]
}

To learn more about the advanced access controls, visit the documentation page.

Then, set the environment variable DELIVERY_STREAM_NAME to your stream name. Once all this is set correctly, the data from the Telemetry API should flow to your destination for further processing. For this example, we’ve set up the destination to an S3 bucket.

If we open one of them, we can see what sort of data that’s flowing through our pipeline.

Great! There are endless possibilities at this point. You can filter the data, process it and query it as your own need. This kind of capability is what excites us and it’s nice to showcase one of them here.

Conclusion

We can see the trend of observability taking over our everyday environment and we’re more than happy to be a part of it, especially when these capabilities are coming to our serverless toolsets. The constant effort of the engineers at AWS to increase the potential of serverless is profound.