6 minutes read

POSTED Dec, 2019 dot IN Serverless

Is it possible to debug Lambdas locally?

Serkan Özal

Written by Serkan Özal


Founder and CTO of Thundra

debug-lambda-locally-1 X

Why would you want to debug your lambdas locally? To answer that question, we need to go back to the basics by asking why local debugging is so highly valued and why debugging is even done in the first place. 

Some developers want to debug to simply get rid of these pesky creatures that inhabit the dark corners of their lambdas. By using a debugger, you make an attempt to understand the origin of unexpected behaviors and remediate their root causes. For other developers, “disinsection” doesn’t quite cut it. Rather than just getting rid of the bugs, these people would like to understand the flow of data in an attempt to better grasp what’s going on. Last but not least, there are the curious minds who would boldly wonder, “What happens if I tweak this?” Having a debugging toolchain in place allows them to extract more knowledge from these experiments, resulting in a higher quality product that is delivered faster.

In this article, we’ll present several tools that may help you with debugging your code. Whether you want to fix unexpected behavior or simply learn more about your data flow, these products can help you.

Test First

Let’s face an uncomfortable truth: email generally works, HTTP performs without a struggle, and AWS is doing just fine; yet, end users keep coming up with bug reports. How come? 

Most bugs happen at the business logic layer and can be thought of as mismatches between the domain model and reality. As development progresses, we tend to focus more on the technical qualities of the code and less on business needs. This underlying issue can be remedied by fighting code with code. In order to satisfy business needs, you should create a suite of automated tests to verify that your software behaves in the expected manner.

Testing is a double-edged sword, though. If you aren’t careful, you risk developing tests that are very hard to write, yet are so brittle that they break during even the tiniest of refactors. Creating these tests isn’t the best use of an engineer's time, and having to do so can be avoided by proper separation of IO and domain code:


def quote_of_the_day():
    quote = requests.get("https://example.com/quote").text

    if len(quote) % 2 == 0:
        return "Your very special quote is: {}".format(quote)
    else:
        return "Your magic quote is: {}".format(quote)

In the above snippet of Python code, the HTTP request is mixed with business logic. Testing it in its current form would require either monkey patching or actually performing the request. The former would make the tests brittle, while the latter would make them slow. However, a simple trick can turn the tables:


def quote_of_the_day(
    request_quote = lambda: 
requests.get("https://example.com/quote").text ):     quote = request_quote()       if len(quote) % 2 == 0:         return "Your very special quote is: {}".format(quote)     else:         return "Your magic quote is: {}".format(quote)

Now, in the test code, you can pass a function that returns the predefined payload and then formulate assertions based on it, while the production code actually makes the request. This idea can be further generalized into dependency injection along with advanced tooling, if needed. These techniques can make testing fun again!

Embrace Function as a Service (FaaS)

You’d like to use lambdas as a service? Then treat them like one! Setting up proper instrumentation will help you to gather application insights both locally and in production. Since the lifespan of a function is short (a maximum of 15 minutes, typically the time of one request), monitoring might not make much sense for individual functions. However, aggregates can provide information about the overall state of the system. A great way to learn about your architecture is to check out the architecture view in Thundra, which you can do in a separate article dedicated to this feature.

On the other hand, centralized tracing and logging (with proper log levels) is absolutely necessary for complex functions. Going through the call graph and lifecycle events will help you discover root causes for defects as well as performance bottlenecks. The tracing and logging tools may also surprise you with their features. Jaeger’s ability to graph the service mesh from spans is invaluable in the case of lambdas, since the connections between lambdas are defined on per function basis and lack a global overview. You can get a global overview by using Thundra’s full tracing capabilities as well.

Debugging: Possible Approaches

There are several ways to approach the problem of debugging. Before digging into tooling, it’s worth checking out the solutions that don’t require additional utilities. You might find that after implementing instrumentation and proper testing, debugging is no longer required.

If debugging is required, you’ll have to decide if it should be done in the cloud or in the comfort of your local machine. Debugging your lambdas while they sit on AWS can be a troublesome process, in part because there’s no debugger there. On the other hand, the downside of running lambdas locally is their lack of integration with the rest of the cloud. In production, this integration is transparently handled by the provider via lambda settings.

From Containers with Love

The most straightforward way to get started with debugging is to use docker-lambda, which spins up a Docker container with your lambda inside. How is this different from putting a lambda in a container yourself? The provided images aim to faithfully replicate the function execution environment. You can take a peek inside the /base folder if you’re curious about what exactly is available to a lambda.

If your code happens to be interacting with other AWS services, you can pass the appropriate credentials via environment variables, and the event can be injected either as an argument to the invocation or through stdin pipe. Upon invocation, a log is printed onto the console with the exact same information you can expect from going through CloudWatch. This UNIX-friendly interface, combined with its perfect coverage of runtime engines, makes this tool suitable for CI. The comprehensive invocation output will help immensely when estimating running time and required resources.

Serverless Application Model

In the context of debugging, the Serverless Application Model (SAM) can be considered an extension of docker-lambda. Yet, the goal of SAM is to provide much more coarse-grained lambda management services via the command line tool aws-sam-cli. It also uses Docker under the hood. SAM handles the whole lifecycle of a function, from development through packaging and deployment. The program works well with CloudFormation. Using the two together, you can provide the necessary backing services for your mesh of lambdas to thrive. SAM integrates seamlessly with text editors like Visual Studio Code and IDEs such as PyCharm or IntelliJ to provide a focused development experience.

AWS on Your Machine

For debugging more complex AWS flows, you might turn your eyes towards localstack. It’s a toolchain for setting up the whole AWS stack directly on your laptop. Once you’ve integrated localstack, the possibilities are endless. In addition to gaining speed from not provisioning remote resources and saving money on cloud bills, you can also set up advanced chaos experiments to test the resilience of your lambda-based system.

Since there is no such thing as a free lunch, there are certain drawbacks associated with using localstack. Configuring it to your preferences might not be as straightforward as it is with the tools discussed previously. In localstack, there are many settings to tweak. Additionally, its parity with AWS is not 100%. At the time of writing, AWS Athena was missing. Bearing these limitations in mind, localstack can be a great aid in your debugging journey.

Debugger on AWS

How about reversing the model a bit, and, instead of running functions locally, bringing a debugger to AWS? With Thundra that’s possible. After a little generic instrumentation, you can direct Thundra’s tracing effort towards the method of your choice. Since it utilizes the environment variables, there is no need to change the code or redeploy the lambda. If you’re looking for a fine-grained approach to tracing, Thundra’s line-by-line tracing is the answer. Below, you can see an example trace chart from a Thundra console showing its line-by-line tracing. It also shows the source code and values of local variables at each line. 

pasted image 0-1

AWS on Debugger

All the other efforts for debugging is simulating the AWS like environment in your local. In these days, we are working on a new way of debugging Lambda functions. We’ll set up a bridge between your AWS account and your IDE (VSCode, IntelliJ IDEA). In this way, you’ll be able to remote-debug your AWS Lambda functions running on their real AWS environment. You’ll be able to see the values of local variables and stacktrace at each line. In this way, you’ll get rid of any concerns of mocking any resources. You can catch more information about this from one of our recent blog posts. This feature is in its alpha stage now but you can still give a try by reaching out to us via support@thundra.io.


Putting the Pedal to the Metal (Almost!)

Underneath every lambda execution, there is a tiny virtual machine (VM) provided by Firecracker, an open-source application. A word of warning: Firecracker is overkill in most lambda debugging use cases, so proceed with caution.

Firecracker can, however, come in handy when parity between the development execution environment and AWS is crucial. It's worth getting familiar with Firecracker’s specifications in order to understand its hard limits in an execution environment. For example, you cannot make your lambda start cold faster than 125ms consistently. This is the time required to spawn a VM.

What do you need to start spinning up micro VMs? One of each of the following items: an uncompressed Linux kernel binary and an ext4 file system image. After having fun with Firecracker, it may be worthwhile to go through its security checklist and witness what happens at the kernel level to carefully isolate each function invocation.

Conclusion

Debugging lambdas locally is by no means a trivial process. The myriad tools and techniques available (including no debugging at all) for doing so can be a bit overwhelming. The wide spectrum of available options does allow you to select the best tool for the job at any system size and FaaS adoption stage, however. Most of the tools have a reasonable learning curve, so try them out, and then stick to the one that fits your use case best.