5 minutes read

POSTED Nov, 2021 dot IN CI/CD

How to Locate Slowdowns in Your CI Pipeline

Serkan Özal

Written by Serkan Özal


Founder and CTO of Thundra

linkedin-share
 X

Development velocity is vital in today’s SaaS world. The faster you can iterate, the faster you can find killer new features for your service.

But this need for speed can work against the imperative to maintain high-quality deliverables. Testing is one way to maintain quality but is often seen as a barrier to new releases. End-to-end tests are especially prone to be slow and hard to debug. If your project has accumulated such tests, you’ve probably noticed your development velocity go down the drain.

So, how can you stay on top of developing new features while maintaining a quality product? Monitoring your CI pipeline will give you important notifications when things slow down. Going a step further, a monitoring tool optimized for CI pipelines can help you locate the culprit of the slowdown and fix it faster than a generic monitoring solution would.

In this article, we’ll demonstrate how to monitor a GitHub Actions CI pipeline with Thundra Foresight, locate slowdowns, and fix them right away.

Prerequisites

This tutorial will use a pre-built AWS CDK project as a base. It requires a GitHub and AWS account, and you’ll need a Node.js installation with npm to run the commands.

The tutorial also assumes that you’re running the examples in Cloud9, a pre-configured cloud IDE by AWS that comes preloaded with all of the required software.

Forking the Repository to Your GitHub Account

You can access our example repository, which has a CDK project, end-to-end tests, and pre-configured GitHub actions. You’ll have to fork this repository into your GitHub account to modify it as well as the actions that run with your credentials.

The repository includes a CDK stack with the following resources:

  • VPC resources for the RDS instance
  • An RDS instance with PostgreSQL
  • A Lambda function with a simple output
  • A Lambda function that accesses the RDS instance
  • An API Gateway to make everything accessible from the outside

Clone the Repository Locally

The example repository includes one GitHub action for running the tests in CI, but the deployment happens from your local machine. This requires you to clone the repository from your GitHub account, which you can do with the following command:

$ git clone <YOUR_FORKED_REPO_URL>

After running this command, you can start browsing the code in your preferred editor.

Install Dependencies

Next, you’ll need to use the command below to install the dependencies so that the deployment can run without interruption. These include packages like TypeScript, Jest, CDK, and others.

$ npm i

After this command finishes executing, you are ready to deploy this stack to AWS.

Setting up Thundra Foresight

Before deploying the infrastructure, you’ll need to set up Thundra Foresight. This is because the Lambda functions are instrumented with the Thundra Lambda layer, and this layer requires the Thundra API key. This is also necessary for getting insights into your GitHub Actions-based test environment.

The example project already includes a GitHub Action in the .github/workflows/run-tests.yml  file, but we have to define the Thundra API key and the Thundra project ID in our repository settings to make it work.

These two keys can be obtained from the Thundra Foresight Web Console. Click on “Create Project,” name it “RDS Example,” and choose the “GitHub Actions” CI integration. This will give you a code example with the API key and the project ID.

Now, open another browser tab for the “Settings” page of your GitHub repository and create two “New repository secrets.”

The first one is called THUNDRA_APIKEY and uses the corresponding API key from the Foresight code example. The second one is called THUNDRA_PROJECT_ID  and uses the project ID from the code example.

You’ll also have to replace the THUNDRA_API_KEY  string at the top of the lib/rds-test-example-stack.ts  file. This ensures that you’ll get the correct trace maps later.

Deploying the Infrastructure

The infrastructure is deployed with the CDK using the command below. The package.json includes a deploy script that also writes an outputs.json  file, which will consist of the API Gateway endpoint URL for our tests. Be aware that this deployment can take over 10 minutes.

$ npm run deploy

After running this command, the application is accessible via HTTP on the URL in the outputs.json.

Running the Tests Locally

To check that everything worked, you can run the tests locally with the following command:

$ npm run test

There are four tests defined, two for every endpoint. The first one executes the cold-start, and the second one runs twenty requests after the Lambda function is warmed up. The index endpoint is handled by a Lambda function that only gives a text output while the database endpoint connects to the RDS instance to read a timestamp.

Pushing the Outputs

Now, we have to get our new outputs.json  file to the remote repository so that GitHub Actions can pick up the URL of our API. We’ll do that with the following command:


$ git add -A
$ git commit -m "Add stack outputs."
$ git push

You then need to commit the outputs.json  file so that the tests can pick up the URL in the GitHub Actions CI later.

At this point, the test run will start automatically, and Thundra Foresight will pick up the results. This run takes a few minutes to complete.

Finding Slow Tests with Foresight

Now, when you look at the Foresight web console, you should find your first test run inside the RDS Example project. If you click on the test run, you’ll be presented with a list of slow tests sorted by the runtime, like the one shown in the figure below.

Figure 1: Slowest tests

If we want to optimize our tests, we can go through this list from top to bottom.

Optimizing the Slowest Test

If we click on the slowest test in our list, we’ll see the time that every run of that test took. This simple example isn’t particularly interesting because it just tells us the test took around four seconds.

However, a click on the “Trace Map” button offers a fascinating view into which part of our infrastructure is responsible for that slowdown. Figure 2 below shows an example trace map.

Figure 2: Trace map

This image reveals that every request took more than 130 milliseconds. Most of this time was spent inside the database Lambda function, and only a fraction of it was an actual request to our database. This implies that the performance problem lies inside our Lambda function code.

We’ve prepared an optimized implementation that only executes init tasks, like loading the database credentials and connecting to the database, once.

To implement this fix, you can replace the content of lib/dbFunction/index.js  with the following code:

const aws = require("aws-sdk");
const postgres = require("pg");

let db;
async function initDatabaseConnection() {
  console.log("Initializing database connection!");
  const secretsManager = new aws.SecretsManager({
    region: "eu-west-1"
  });

  const dbSecret = await secretsManager
    .getSecretValue({ SecretId: process.env.DB_SECRET_ARN })
    .promise();

  const dbCredentials = JSON.parse(dbSecret.SecretString);

  db = new postgres.Client({
    host: dbCredentials.host,
    port: dbCredentials.port,
    user: dbCredentials.username,
    password: dbCredentials.password,
  });

  await db.connect();
}

exports.handler = async (event) => {
  if (!db) await initDatabaseConnection();

  const result = await db.query("SELECT NOW()");

  console.log(result);
  
  return {
    statusCode: 200,
    body: JSON.stringify(result),
  };
};

After you have updated the file, rerun the deployment with the following command:

$ npm run deploy

If you commit the changes and push them again, the test will be rerun with GitHub Actions.


$ git add -A
$ git commit -m "Improve performance"
$ git push

Figure 3: Updated trace map

Figure 3 shows how our time inside the Lambda function went down to just 18 milliseconds. Because Lambda is billed by millisecond, this is better for us and our clients.

Conclusion

Development velocity is an important component of innovation, but it shouldn’t come at the expense of code quality. That’s why monitoring your delivery pipeline is essential—and using a monitoring system that has been tailored for test CI monitoring is even better. With such a specialized tool, you can quickly find and fix delivery problems for better products at a lower cost.

Thundra Foresight is just such a tool. It works with the popular JavaScript testing framework Jest as well as GitHub Actions. It also delivers additional features like trace maps, so you get deeper insights into your test infrastructure for painless remediation.

Gain visibility into your CI pipelines with Thundra Foresight so you can find and fix slowdowns with ease.