5 minutes read

POSTED Dec, 2020 dot IN Microservices

Monitoring Microservices on AWS with Thundra: Part I

Serkan Özal

Written by Serkan Özal


Founder and CTO of Thundra

linkedin-share
 X

Thundra is a monitoring service built from the ground up for serverless microservices. Thundra can map out your whole architecture, but it also lets you drill down to particular Lambda functions that contain the custom code that glues your whole system together.

Thundra even goes so far as to allow offline and online debugging of your Lambda functions, so you can find bugs right inside your production environment without any mock servers or local development deployments, which eases error replication a great deal.

While these claims all sound nice on paper, what does this mean in practice? After all, serverless and microservices aren’t terms that describe a specific technology, but a whole range of technologies.

In this three-part article, we will answer that question by building three systems based on different serverless technologies. The first uses AWS Lambda at its core, the second will use Kubernetes, and the final system will use Amazon ECS. The container examples will run on AWS Fargate to make them as serverless as possible.

Let’s dive into the first example based on AWS Lambda.

Example Project

The first example project is an asynchronous factorial calculator, called heavy-computation. It receives a number and an email address via HTTP, calculates the factorial of the number asynchronously, and sends the result back to the email.

Heavy computation architecture

Figure 1: Heavy computation architecture

Figure 1 shows the core architecture diagram: three Lambda functions, two S3 buckets, one API Gateway, and an SES as email API.

We will use the AWS CDK as our “infrastructure as code” (IaC) framework. The CDK will create additional AWS resources that are missing in this manually created architecture diagram to support functionality.

Note: The project has two bugs in the Lambda code, and we’ll try to fix them, first with AWS CloudWatch and then with Thundra.

Prerequisites

The following prerequisites are required to install this project:

  • An AWS account
  • A Thundra account, with your AWS account set up
  • AWS CDK
  • Node.js
  • NPM

Installing the Example Project

Clone the Git repository to your local machine:

$ git clone https://github.com/thundra-io/heavy-computation

Install dependencies with the following command:

$ npm i

Replace <SOURCE_EMAIL> in lib/email.json  with an email address you have access to.

Add this email address to the verified email list of SES in the AWS Console and click the link you get within the activation email.

Note: For security reasons, Amazon SES is in sandbox mode by default, which only allows you to send emails to verified addresses. Sandbox mode can only be deactivated by writing a support request to AWS.

Deploy the project with these commands:

$ npm run bootstrap
$ npm run deploy

After these commands complete, the example project is deployed on AWS and ready to calculate factorials.

Using the Example Project

After we deployed the project, the CDK showed us the endpoint URL of our new HTTP API. We can use this URL to send an example request to it with cURL:

$ curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"email":"<EMAIL>","number": 10}' \
       <URL>

Replace <EMAIL>  with the email you verified for SES and <URL>  with the output from the CDK command.

The response should be {"message": "Internal server error"}  because some Lambda functions have bugs right now.

Debugging with Amazon CloudWatch

To find our bugs, we can look into the CloudWatch logs of our Lambda functions. If you want to brush up on your CloudWatch knowledge, or you don’t know much about it in the first place, this article will be useful to read.

The backend function is the first Lambda function in our pipeline. We have to search for it inside the AWS Console. It should have a name that starts with “HeavyComputationStack-backend.”

  1. Click on the function name.
  2. Click on “Monitoring” on the top left.
  3. Click on “View logs in CloudWatch” on the right.
  4. Click on the top “Log Stream” at the bottom.

This leads us to the error log of our backend function.

Backend error log

Figure 2: Backend error log

Figure 2 shows the backend error log. We can see that a parameter named “Key” wasn’t of type string in our backend function.

To fix this, we have to replace line 14 in lib/functions/backend/index.js with the following code:

Key: Math.random() + "-" + Date.now() + ".json",

The new code will convert the number to a string ending with .json, so the AWS SDK won’t complain anymore.

Let’s deploy the changes with this command and see if it worked:

$ npm run deploy

If we run the cURL command after this deployment, we should get an OK as reply, but still no email. That’s because there is one bug left!

If you navigate to the CloudWatch logs from the email Lambda function, you can see the next problem: "Unexpected key 'ToAdresses' found in params.Destination"

There is a typo in the email definition of our Lambda function. One “d” is missing.

To fix this, we have to replace line 32 of our lib/functions/email/index.js  file with this code:

Destination: { ToAddresses: [destination] },

Save the file, deploy, and run the cURL command again! Now we are finally getting our email.

Debugging with Thundra

To debug our system with Thundra, we first need to reset the example project. To do this, run the following command in the project directory:

$ git reset --hard

We then have to replace SOURCE_EMAIL inside the lib/email.json with the email address we validated for SES.

Then we have to instrument our Lambda functions by adding the Thundra Lambda layer to it. This can be done by replacing the createLambdaFunction method of our HeavyComputationStack class in the lib/heavy-computation-stack.js file with the following version:

 createLambdaFunction(name, environment = {}, role = null) {
  if (!this.thundraLayer) {
    this.thundraLayer = lambda.LayerVersion.fromLayerVersionArn(
      this,
      "ThundraLayer",
      "arn:aws:lambda:" + process.env.CDK_DEFAULT_REGION + ":269863060030:layer:thundra-lambda-node-layer:48"
    );
  }
  environment["thundra_apiKey"] = "<THUNDRA_API_KEY>";
  return new lambda.Function(this, name, {
    runtime: lambda.Runtime.PROVIDED,
    handler: "index.handler",
    code: lambda.Code.fromAsset(
      path.join(__dirname, "functions", name)
    ),
    environment,
    layers: [this.thundraLayer],
    role,
  });
}

Replace <THUNDRA_API_KEY> with your own key, which you can find inside the Thundra Console.

Deploy again with the CDK:

$ npm run deploy

Then run the cURL command to see the first bug. Again, you have to replace the <EMAIL> with the one you validated for SES and the <URL> with the URL output from the deploy command.

$ curl --header "Content-Type: application/json" \
  --request POST \
  --data '{"email":"<EMAIL>","number": 10}' \
       <URL>

Now that we see the {"message": "Internal server error"} response again, we’ll have to investigate, but this time we can look into the Thundra Console.

If you navigate to the Functions page, you’ll see the backend Lambda function at the top, with at least one invocation and error. Well, this was much easier to find than in CloudWatch. Clicking on the function name will lead us to the “Function Details,” which show us all invocations of that function.

Backend Lambda invocation trace chart

Figure 3: Backend Lambda invocation trace chart

Since we instrumented that function with the Thundra Lambda layer, we can see that the invocation was triggered by API Gateway and that it had an “InvalidParameterType” error. If we click on that erroneous invocation it will show us that two services were used in the invocation, Lambda and S3. If we click on S3, which also shows us that a write operation happened, we see that the error message is "Expected params.Key to be a string.” Figure 3 shows us how the invocation trace chart would display that error message.

To fix this, we replace line 14 in the lib/functions/backend/index.js file with the following code:

Key: Math.random() + "-" + Date.now() + ".json",

We redeploy, use cURL again, and are off to the Thundra Console to spot the next error.

This time all of our Lambda functions were called, but the email function had an error, so let’s click on it! In the Function Details we see the erroneous invocation and also click on it.

Email Lambda invocation trace chart

Figure 4: Email Lambda invocation trace chart

If we look at the trace chart in figure 4, we find our typo through the error message “Unexpected key 'ToAdresses' found in params.Destination.”

To fix this, we replace line 32 in the lib/functions/email/index.js file with the following code:

Destination: { ToAddresses: [destination] },

Redeploy, use cURL, and look into the Thundra Console again. This time the invocation count of all of our Lambda functions should be higher, but the error counts should be the same. We have our email.

What We Learned from Debugging with Thundra

In this article you learned how easy it is to integrate Thundra with AWS Lambda and how it helps to find bugs.

The Thundra Console is first and foremost about Lambda functions. Just navigating to the Functions page reveals what Lambdas there are, how often they were called, and if they had any errors. In comparison, the AWS GUIs for Lambda and CloudWatch are rather convoluted, not intuitively labeled, and often lack in terms of performance.

Getting a third-party monitoring service integrated into your AWS microservices should be the first step when you start a new project, and Thundra is an obvious choice.

In the next article, we’ll talk about Thundra and containers, specifically with Kubernetes on AWS Fargate. Stay tuned!