Debug AWS Lambda Node.js Functions in Production Without Code Change

We introduced rich support for Node.js applications previously. In this blog, I want to go into our support for manual and automated instrumentation and show you how to add this to your Node.js applications. But, first, let’s talk about why instrumentation is useful in serverless environments...

Instrumentation is a term generally used for scientific experiments. Scientists always measure something to find the answer to the questions.  For example, Is an increase in gas pressure correlated to an increase of its temperature? Or, does cyanide oxide affect decrease the number of microorganisms in a petri dish? Or, what is the distance between Alpha Centauri and Sun? Scientists conduct experiments and compare results to learn more about the world around them.

We, software engineers, are not far away from the scientists in experimenting. Our world is our production environments. And we need to constantly measure the performance of our applications running on them in order to keep the application running optimally.

This mindset is also true in serverless environments. You notice an individual function calls taking too long to execute. Then you go pinpoint where the latency problem is occuring and try to understand why it’s happening.

Instrumentation allows you to measure the behavior of your AWS Lambda functions in the form of spans. Spans provide detailed insights into the behavior of your functions, including information such as which function calls upon another function, the function arguments, return values, and possible errors. Moreover, it also provides information about external calls your invocation makes, such as to your Redis caches, AWS services, HTTP endpoints and much more.

Manual Instrumentation with Open Tracing

With manual instrumentation, you can add custom spans to your code.  This lets you measure the execution time of your functions, any third party library, or any external services (such as API or database calls) and then visualize that execution time within the Thundra console using a simple programmatic API.

The Thundra Agent is OpenTracing compliant, and thus you may use the OpenTracing API and the Thundra Tracer to measure execution times for specific pieces of code. With the help of OpenTracing, you can have end-to-end visibility of your distributed system, including your AWS Lambda environments..

However, manual instrumentation does require you to make code changes. If you have a large enough system, you probably do not want to change all your code and break it up.  Therefore, manual instrumentation is better employed if you plan for it from the beginning by implementing it as you write your code.

For larger systems or for when you want to extract information from your system without needing to manually change code, we suggest using automated instrumentation, which  Thundra now supports for Node.js applications in AWS Lambda!

Automated Instrumentation

Automated instrumentation gives us the same information about our functions but without requiring any code changes!

Automated instrumentation is a perfect fit for AWS Lambda because not only you can benefit from zero code changes while implementing instrumentation but also you can configure which functions to instrument by only changing the environment variables. That means you do not need to redeploy your functions. This is particularly useful in production environments when things go wrong and you want to instrument a suspicious piece of your Lambda function to solve the “murder mystery”.

Diving into Node.js instrumentation

Let’s now go deeper and see what you can do with Thundra’s Node.js instrumentation support.

In our example, we have a simple DynamoDB table named Avengers. The table includes hero names and we register additional names into our database according to incoming requests. If the hero is already enlisted, we just return the response.You can check out the code in our GitHub repository.

When you use Thundra’s automated instrumentation, you first need to configure which methods would you like to instrument. In the following example, we have the  `getHero` method in the `services.js` file. We initialize our Thundra agent with the following configuration:

var config = { 
	traceConfig: { 
		traceableConfigs: [{ pattern: 'services.*' }], 
	}
}

We immediately get useful information, as seen in the screenshot below.

Here we see the order in which each function executes. If we click on a function, we can also see all arguments, return values, and errors (if they occur).

Thundra now enables AWS integration by default with version 2.0.5. This means that you can easily instrument your DynamoDB calls. When we do that (as seen below), we see that our Lambda function made a call to scan the DynamoDB table. We can also see the call’s duration. Clearly, this is the reason for our latency in our Lambda function’s invocation.

One thing to notice is that the getHero method making the  dynamodb.scan call lasts shorter than the dynamodb.scan call, itself. The reason is that Node.js executes these invocations asynchronously. This is one reason why Node.js is so popular and powerful but also a reason why developers like me can find Node.js frustrating to work with. It can be difficult to understand execution order of the various functions.

What if we tried to get this information via manual instrumentation?  If we tried to instrument the same code with manual instrumentation, we would need to start a span and finish it in every function.  Here’s an example of what that would look like:

const tracer = thundra.tracer();
const span = tracer.startSpan('getHero'); 
heroServices.getHero(event).then(
    … 
    span.finish(); 
    callback(null, "Hero has already been enlisted");

With this approach, we are forced to close the span before calling each callback. The resulting trace charts would like this:

Unfortunately, we can’t see arguments and return values.We would need to add them manually as well. This approach requires a lot more work for the developer and does not give us the most accurate information about what execution is happening when and how.

Therefore, our recommendation is to always favor automated instrumentation except when you want to add custom data on your spans.  For custom data, such as business related metrics, manual instrumentation is very useful.

Conclusion

Automated instrumentation is a perfect fit for serverless development because you can benefit from getting accurate and valuable information without making any changes to your application code. All you need to do is import Thundra core libraries and configure which functions to instrument by using environment variables. Want to try it out? Register for a free trial at https://console.thundra.io/signup .

And, would you like to learn more about our Node.js integrations?  Stay tuned in the following weeks and we’ll be diving into some of our integrations and how you can use them to learn about what’s going on in your serverless environments.