We have great news. Now you can use your own ELK stack with our freshly baked Thundra plugin. We know that for some of you it is really important to keep every bit of the data on your safe environment. Here at Thundra, we want serverless developers to be as free and safe as they can. Our true aim is instead of just providing the rapidly evolving Thundra SaaS application, we prefer offering a package of integrations with our core technology on top of our application. We provided our Splunk and Honeycomb integrations solely for this purpose and today we are proudly releasing a third, ELK integration.
We want to offer serverless developers a hassle-free experience on their journey. Because we know serverless is easy to start and as easy to mess up later. It is small, tiny and even cute when you first implement your `Hello World` example, and then you make your first invocation to it. If you are caught up and got used to that magic, it is inevitable that your baby becomes a juggernaut which starts spitting data on your S3, syncing data between Dynamo and Redis, sending emails or push notifications to your customers. This is serverless, everything evolves fast, architectures tend to blossom. At this very moment, Murphy comes to a visit on your Lambda, because he always does, and everything goes wrong inside your black sweet box of Lambda. You start wishing for the hammer that could just break the Lambda’s box for you. Your wishes come true, Thundra is just the right tool that can show you the weaknesses and interest points inside your serverless juggernaut from a bird’s eye view to the tiniest detail.
There's no rose without a thorn. Serverless is something that produces a mass amount of data. Saving it, storing it, moving it, analyzing it is always a problem through the daily routine of mere developers. Do you know what is more dreadful about data, yes I heard you, securing it. Even though we are really sensitive about keeping our customers’ data in the most secure way, we really understand them when they are hesitant about sending their values to us. This is why we also provide some good integrations to the most popular platforms like Splunk for enterprises and Honeycomb for data visualization enthusiasts and now ELK plugin for generalists. Because who works on data without a need to search over it.
Thundra ELK plugin is great if you are using ELK stack for your development post-mortems and want to benefit from Thundra’s wonderful tracing and monitoring capabilities. We offer you this privilege in 2 steps.
1. Setup your Thundra Integrator for Elasticsearch
First, you need to contact us by sending an email to email@example.com to obtain a license key to use Thundra Integrator to send data to your EC2 instance. After that Thundra provides an AWS CloudFormation template to easily set up your integration. Below is the deployment topology created from the CloudFormation template. It includes:
- The Thundra Receiver App that collects, transforms and aggregates data from the Thundra Lambda agent.
- Filebeat that sends that data to your Elasticsearch.
- A data volume that temporarily stores the files and data collected from the Thundra integrator. This data volume cleaned by the Thundra integrator hourly. Your EC2 instance should look like something similar to below when you are done.
You can find more detailed information about how to set up your Elasticsearch AMI on your AWS EC2 instance for Thundra integration here. If you are already using Thundra’s agents to send data to our beloved web app, you are almost done. You just need to adjust your Lambda environment variables from the configuration accordingly.
When you setup all above, you directed the data flow to your new EC2 instance. Thundra agents will hand in the data to the Thundra integrator, and it saves it to your new Elasticsearch instance over EC2. From now on, you will not see any new invocations on Thundra console app; instead, you need to search for your data on Elasticsearch using Kibana. If this is not so practical, throwing away our swiss-army knife web application, we offer you a Thundra Kibana plugin which can replace our web application in terms of most common use cases like looking for traces, logs, and metrics. We will explain our Kibana plugin as the final step to your data privacy.
2. Thundra’s Kibana plugin
We provide a Thundra Kibana plugin, using which you can visualize relevant data for all your Lambda functions. We distribute Thundra Kibana plugin in 2 ways. The first way is you can download our plugin build using this link. Then you can install it as if any other Kibana plugin using this command
bin/kibana-plugin install <package name or URL>. The second way is we provide our plugin code as open source over github. So you can use our source code to create your own builds and integrate Thundra Kibana plugin to your own environment.
Thundra’s Kibana plugin is a gist of the main application that we serve to our users via our console app. You can follow this link to set up the plugin for your environment. The plugin basically provides these features;
- List all your functions
- List all invocations of a function
- Shows meta information of functions and invocations
- Show traces and spans of an invocation
- List logs of an invocation
- Show metrics for a function
You can get a more in-depth idea about how to use these features by following our documentation. Once you are all set you can see Thundra logo on the sidebar of the Kibana as shown below.
When you need to learn more about your functions, you can see the invocation details with trace charts just like you can achieve in our SaaS environment. In the following image, you can understand that reaching out DynamoDB is causing the slowdown and you can see the query sent to the table.
When you need to learn about the logs of your invocation, it is also achievable with our Kibana plugin. In the following image, you can see the logs from different log levels of a single invocation.
Using Thundra’s Kibana plugin, we support a unique way of monitoring your Lambda functions inside your own nest. This brings two great benefits within. First and foremost, we become liberated from dealing with invaluable customer data; second, it resolves any security concern by keeping any telemetry or log value inside our users' self-managed EC2 instances. As Thundra, the leading monitoring and observability solutions for AWS Lambda, we are always looking for the best platform that suits our users helping them deeply understand their Lambda functions. The latest result of our customer success story is our Thundra ELK Integration. If you would like to see any other integration that you think would be beneficial, please just send an email to firstname.lastname@example.org.
You can sign up to our web console and start experimenting. Don’t forget to explore our demo environment — no sign up needed. We are very curious about your comments and feedback. Join our Slack channel or send a tweet or contact us from our website!