5 minutes read

POSTED Jan, 2020 dot IN Observability

A Step Towards Observability with Lambda Destinations

Sarjeel Yusuf

Written by Sarjeel Yusuf

Product Manager @Atlassian


If we are to loosely extrapolate the well-known Chinese proverb, “a journey of a thousand miles starts with a single step”, we can build the story of how AWS Lambda Functions has matured on its journey to an ideal serverless solution. The absolute optimization of serverless, where all known barriers are overcome, may still be far, but AWS has been industriously taking the steps to this goal. One of the latest being AWS Lambda Destinations.

Released in late November, just a few weeks before AWS re:Invent 2019, AWS Lambda Destinations aims to better improve asynchronous invocations. Additional benefits include improved observability of your AWS Lambda invocations and reduced code complexity in handling asynchronous invocations. This new feature, that many may deem insignificant compared to more celebrated improvements such as Provisioned Concurrency, is actually a big improvement in how we execute asynchronous invocations with AWS Lambda. 

The question thus is, how does this impact development with AWS Lambda functions and what were the prior alternatives? That is the purpose of this piece. To introduce the concept of AWS Lambda Destinations, how it betters development with AWS Lambda and how we can practically put it to use. 

The Motivation for Lambda Destinations

AWS Lambda functions, and in fact serverless in general, is notorious for its black-box effect. One of the drawbacks of working with serverless functions is that we do not know what is going on under the hood and just expect either the right or wrong output.

Of course “not knowing what goes on under the hood” is a generalized statement, but overall observability is a major issue with the serverless model. The origins of the problem stems from the same source of serverless’ major advantage. The property of being fully managed. Using serverless functions is extremely easy as there is no orchestration required and all responsibility is delegated to the cloud vendor, and this is the source of the black-box.

The list of exact issues that contribute to the serverless black-box problem is truly a long one. However, cloud vendors are taking steps toward improving the situation. These efforts are coupled with the services of third party SaaS observability tools such as Thundra.io. The overall result is an eased serverless development experience but nevertheless the inherent problem of serverless still does exist.

This problem is further exasperated when the vendor’s own limitations are stacked over the conceptual problems of serverless, and that is what AWS is tackling in its journey to being the top serverless compute service provider. AWS Lambda Destinations is one of these improvements, acting as a step in the right direction. 

Introducing the Destination

The problem being addressed by Lambda Destinations is one related to asynchronous invocations of AWS Lambda functions. When a Lambda is invoked an event is sent to AWS’ internal queue. Upon successfully being put into the event, a 2xx status code is returned. This is all well and good until it is realized that apart from the status code, there is no additional information that is received. This is because it is the queue itself that returns the response code.

Consequently, we do not actually know if the Lambda function was invoked successfully asynchronously or if it failed. Hence the lack of observability of async invocations which also adds to the second problem. This issue relates to the control of workflows we can achieve with AWS Lambda.

Considering that AWS Lambda services are most commonly used in building microservices in the cloud, owing to its nature of operation, the lack of observability hinders our efforts of building these serverless microservice applications. This is because the general idea of microservice architectures is that individually decoupled components communicate with one another depending on the events being passed along the workflows. 


Therefore, one of the expectations of using AWS Lambda functions is that it can communicate with other components within the configured architecture, preferably with different events upon failure or success of the Lambda function invocation. This is where the second problem, building upon the first becomes apparent.

Depending on the status of the async invocation, the Lambda function would be expected to send different events to different targets. In the previous model of AWS Lambda, these communication needs would have to be programmatically configured when writing the code. For example, a Lambda function would be required to trigger another AWS Lambda function of success, or send an event to AWS EventBridge on failure. 

Since this control logic would be handled within the AWS Lambda code itself, it would eat away from the available 15-minute maximum runtime that AWS provides. Trying to perform such workflow logic with the timeout restriction becomes an inimical pattern to the success of the serverless application. 

This is where AWS Lambda Destinations comes in providing respite from all the issues discussed. Configuring Destinations means that results from Asynchronous destinations can now be routed to a destination resource depending on the status of the invocation. These results are execution records in the form of JSON objects containing details about the request and response of the invocation. As a result, these execution records enhance observability within the serverless application. 

Moreover, with the novel feature allowing you to configure routing directly, you no longer need to add the infrastructure connections within your ‘onSuccess’ or ‘onEvent’ control blocks. Instead, you can go directly to the console and add target destinations using the ‘Add Destination’ button. This is shown in the figure below:


By overcoming the need to configure destinations within the AWS Lambda code itself, AWS has promoted brevity of code. Taking advantage of the new feature, developers no longer need to manage routing logic and instead can focus primarily on their business logic. Truly a step forward to providing a fully managed service. 

Embarking Towards the Destination

As expected, configuring AWS Lambda destinations can be done in various ways and not only through the AWS Lambda console. Destination configurations can also be achieved through  AWS CLI, AWS SAM, AWS CloudFormation, and event via some language-specific SDKs for AWS Lambda functions.

Irrespective of the method chosen to perform the configuration, the business logic upon routing would follow the same structure, and could be as simple as the code snippet shown below:


As can be seen in the code snippet, no infrastructure is defined within the routing logic and only the business logic is included. AWS knows which service to route the service to depending on your configuration of destinations. As shown in the previous images, configuring these destinations can be as easy as clicking on the ‘Add destination’ button in the Lambda console and simply select from your existing services depending on the status of the event.

As noted, the AWS Lambda console has undergone a change, manifesting the introduction of the new feature. For example, ion the position of the ‘Add destination’ button, the console would display the services which the Lambda function had permission to access. This information has now been put under the ‘Permissions’ tab.

The process of setting up the trigger, however, has remained the same. Therefore in setting up the trigger and triggering the AWS Lambda function asynchronously, the right destination service can be expected to be triggered according to the configurations made. These can be validated using AWS CloudWatch will capture the execution records. 

Senior developer advocate, Julian Wood, provides a similar practical walkthrough on his blog introducing AWS Lambda destinations. To get up and running it is recommended to check out his blog piece.

Lessons Learnt Reaching the Destination

Accounting for the observability concerns that plague the world of serverless, AWS Lambda Destinations is definitely a welcomed improvement for the serverless community. With the novel feature, developers are now out of the dark possessing the knowledge of their asynchronous invocations.

Building on top of the improved observability capabilities, AWS Lambda Destinations has also eased building of intended workflows by simplifying configuration of destination resources, depending on the status of the asynchronous invocations. That means developers now have to worry less about setting up the infrastructure in their code, truly moving AWS Lambda a level up in terms of managed services.

However, we are not out of the woods yet. AWS Lambda, and the serverless model in general still has many observability concerns to deal with. Yes, Destinations has made it easier to set up the serverless application infrastructure, but still has not removed the observability issue around visualizing the infrastructure, among other concerns. This is where coupling AWS Lambda development with third party tools, such as Thundra.io mentioned previously, provides the edge needed.

In conclusion, there is no doubt that AWS Lambda Destinations will greatly improve the development experience. The release of the feature simply delineates AWS’ endeavours of improving cloud computing, and more specifically the niche of serverless.