EventBridge Schema Registry and How to Use It

Mar 19, 2020

 

Eventbridge-schema-registry


EventBridge Schema Registry

EventBridge is a serverless event bus launched to connect applications together using data from SaaS applications and AWS services. EventBridge enables SaaS applications, such as MongoDB, Thundra, and Zendesk, to send real-time data to AWS services like Lambda, ECS, and so on. EventBridge lets you build an event-driven flow very easily by taking care of event ingestion, delivery, error-handling, and security.

Before the launch of the EventBridge service, there was no event bus available to enable a direct connection between SaaS vendors and AWS. Developers used to try Polling or SaaS webhook patterns to send data to AWS services, but these patterns had limitations and were also complex to manage. 

EventBridge was launched in July 2019 and was built using the existing event-processing model that had been used as a base for CloudWatch Events. It therefore has a default event bus inherited from CloudWatch and accepts events from AWS services. It also supports the SaaS event bus that accepts events from registered AWS SaaS vendors. 

Here, in this article, we’ll discuss two new EventBridge features, Schema Registry and Discovery, launched via “Preview” status. But first, let’s discuss the three major components of the EventBridge service.

Partner Event Sources

Event Sources are SaaS platforms, custom apps, or an AWS service. On a SaaS platform, when we register the AWS account, it will create an event source in the AWS account that will be associated with an event bus. To secure this registration process, users have to accept the invitation sent by the SaaS partner in AWS; after this, the event source becomes active and can accept events.

Event Bus

In EventBridge, there are three types of event buses:

  • Default event bus: Every account has a default bus, which accepts events emitted by AWS services.
  • Partner event bus: This can receive events from applications hosted at AWS SaaS partners. For each SaaS partner, a separate event bus needs to be created.
  • Custom event bus: This type can receive events from custom applications and services.

Event buses can have a maximum of 100 rules associated with it.

Rules

When an event comes to an event bus, it needs to be routed to a target service. This is decided by predefined rules. A rule has to first define a pattern, which can also be customized manually, in which the developer needs to select an event source and event type. Then, it has to select an event bus and target. A single rule can point to multiple targets and will be triggered in parallel. A rule can filter and transform the event before routing to target services.

Process of Event Discovery

EventBridge was introduced to ease the development of event-driven architectures and allows applications to decouple from each other. Developers write independent code connecting to an event bus, so no point-to-point integration service is required. However, there is the challenge of finding events and the schema of the event.

Current Manual Process

Every event is a data payload. It may have several fields in the form of json or any other data structure. Currently, producers publish the event schema in a document, which developers need to go through to understand. It is a manual process to find this schema and then create an object in the code manually for that event. This whole process is clumsy and prone to errors. You also need to go through multiple attempts to fix any errors and get the event working, which adds to the cost and time required to deliver.

Schema Registry and Discovery

Schema Registry, used to store a collection of schemas, is a new feature introduced in EventBridge to address the manual registry process. Schema Discovery automates the process of searching schemas and adding them to the registry. Once enabled at the event-bus level, a schema is automatically generated based on the event being sent to the bus and stored in the registry. Any change in events automatically creates a new version of the schema in the registry and provides an option to generate the code binding for the schema. Schema Discovery is generally enabled in a test environment to get the schema information. However, it can also be used in a prod env simply to audit the events.

Code Bindings

As mentioned earlier, once a schema is created, it provides the option to download a code binding. This helps to create a strongly typed object in the code to create events and perform validation before sending. 

EventBridge code binding–available for Python, Java, or TypeScript–can be downloaded through AWS CLI, a console, or an IDE. For an IDE, like VSCode or IntelliJ, you need to use an AWS Toolkit plugin. The AWS SAM framework also supports schema-generation features.

Common Use Cases

Let’s walk through some of the common use cases where EventBridge and a schema registry can be used. 

Security Operations

Securing an application on an AWS infrastructure is a shared responsibility. This means that the burden is on the application owner to log security events. 

A SaaS solution provider that integrates with AWS services can retrieve threat detection events and build a communication workflow, as well as send reports back to AWS using event-based rules. These events can automatically generate a schema at an event bus once real-time events flow through. Then, AWS services like Lambda can use the code binding, generate the event object, and send the notification (using an SNS service) to security and application teams for them to take action as required per the given threat event.

Applications Monitoring

There are many SaaS vendors who integrate with AWS to receive real-time application metrics related to monitoring. Metrics such as CPU utilization, load volumes, and error rates, can be used by vendors to generate alerts and send them back through EventBridge to trigger a Lambda Function. 

Alerts can be of multiple types. When they are sent to an event bus, it automatically generates a schema and has a section for each type of these alert events. Through code binding, objects can be created to filter different types of alerts and take action accordingly. For example, if CPU utilization is high, rules can be defined to connect to CloudWatch events to scale the EC2 instances. If error rates are high, it can send a notification (using SNS) to the support the team to take action for recovery.

CRM Integration

Every enterprise has a CRM tool to manage its user information, and vendors like OneLogin integrate with AWS services using EventBridge. An event of a new-user creation is sent via an event bus to a Lambda Function that can spin up the required resources for the newly added engineers on the team. 

Again, the event schema plays a major role, as there can be many types of resources and you need to filter a user depending on its role and department. Automatic generation of schema helps to build the event object based on live events, and rules can be written to filter those events and send them to the relevant target services.

Performance & Security Observability

Performance monitoring and security observability are very important aspects of application monitoring. Especially for serverless applications, end-to-end visibility is a must. 

Thundra traces serverless event-driven apps end to end via distributed tracing. It generates rich metrics data and transforms it into actionable insights and can also integrate with EventBridge, allowing customers to remediate performance and security issues. Once alerts are sent to EventBridge, AWS Lambda or any other AWS resource can be triggered to use the event information in order to immediately execute the required response and remediate the issue.

Take a Lambda Function that has a logic to execute within 100 milliseconds. If it crosses that limit, it is going to impact the user experience as well as cost. You need to enable an alert mechanism to send notifications in case the execution goes above this limit.

  • You would need to observe the logs metrics and find the ones that cross the 100 ms limit. A sample of the logs would be like this:

REPORT RequestId: 0858f0d2-8e67-4150-b705-7c1b4329192f

Duration: 177.70 ms Billed Duration: 200 ms Memory Size: 2048 MB Max Memory Used: 141 MB


  • Using Thundra’s plugin, the logs can be emitted from a Lambda service to Thundra. An alert policy can then be written using the query (AVG(Duration)>100) to identify the issue and raise an alert. 
  • This policy will generate alerts and send them back to the  target Lambda service.

A sample of an alert event:

{
“version” :0 ,
“id” : “aadfdsfj-2e22-d23f-2se9-awek09bs1c”,
“detail-type” : “Thundra-Event”,
“source” : “aws.partner/thundra.io/3fdf2sd2-thundra-event-source”,
“account” : “XXXXXXX”,
“Time” :
“region” :  
“resources” :
“Detail” : {
    “severity” : “critical”,
    “summary” : “ A thundra alert event by Alert policy.”,
    “thresholdOperation” : “greater than”,
    “periond” : “5 min”,
    “alertPoilcyName” : “Critical alert”,
    “query” : “AVG(Duration)>100”,
        “alertEventLink” : “https”//console.thundra.io/alert-event/alert-event-9kj2io2u-2dw4-1sd3-98l23hc2ff2”,
        “resultNumber” : “1”
        }
}
  • In EventBridge, there is a section named Schema registry and schemas. Based on the above event, it will generate the schema and code bindings. Navigate to that schema, and download the code bindings. 
  • This codebinding is actually an openAPI 3.0.0 spec. The best way to use this schema is to integrate it with an IDE using AWS Explorer. If we build a serverless Lambda function, a SAM option is available under AWS Explorer. It will help generate a Lambda handler method by selecting the discovered schema for the above event (Discovered Schema Registry section in the AWS console).  
  • Now, in the Lambda handler, write the logic to filter such alert events, spin down that particular instance, and send an SNS notification to the Security team with the information related to the given user to take further action.

Many such real-time scenarios can be solved using this flow. The automatic discovery of schema reduces the pain of manually creating the event object and avoids potential manual errors.

Pricing

There is no cost associated with using a schema registry.

For Schema Discovery, 5 million ingested events per month are free under the free-tier plan. This is a very high limit and should be enough, as it is generally used only in a non-prod environment during the development phase.

If you do cross the free-tier limit, then EventBridge charges $0.10 per every million events ingested for discovery, measured in 8 KB chunks.

Summary

To conclude, EventBridge is getting more and more popular because it is a simple and robust service that enables and builds event-driven flows in AWS. It’s also constantly adding new SaaS vendors to grow this ecosystem. In addition, automated schema generation and discovery helps resolve many real-time use cases, as discussed in this article, adding even more value and ease-of-use to this service.