Reel in troves of data with webhooks

When we ingest security data at Red Canary, we need it to be reliable and complete. Some providers offer data via callback requests called webhooks. They are convenient, but as with any push-based notification, we need to make sure that our systems are always ready to receive the event and that all received events are correctly processed. Any interruption, error, or problem could result in lost data.

How Red Canary uses webhooks

The web was designed for clients to request data from a server and get a response. The response could be an error code of some variety or some content to display.  The client is responsible for resending the request if there’s an error. Webhooks extend that popular protocol to new areas. When new data arrives, a client will make a request to a web server Red Canary operates telling us about the event. There are no standards or specifications for what happens if that request errors, times out, or fails for any reason, which makes it extra important that the server always captures the event and responds with success to the client.

As an example, consider events notifying the webhook receiver that there’s a new file available for download. The files may be varying sizes, compressed, and include multiple records. Fully processing the request will take an unknown amount of time to complete, longer perhaps than the client timeout. The safest, most reliable move is to receive the webhook event and store it for later processing so that you can promptly respond with a success to the webhook client. This sort of asynchronous processing is a common pattern in web development and requires a job system backing the web server along with a second system to process the jobs.

Asynchronous processing is a common pattern in web development and requires a job system backing the web server along with a second system to process the jobs.

With a background-processing system we can quickly respond to webhook notifications and reliably process them, but only if our web server is running and available to receive and respond. If the web server is unreachable, we’ll never know we missed a message and we may not ever see it again. Like in the above example of long-running requests, there are many known patterns for ensuring reliability, including multiple servers, geographically distributed deployments, and phased rollouts—all with ever increasing amounts of complexity and cost in exchange for greater resiliency. So how do we process webhooks with low complexity, low cost, and high resilience?

An API gateway to success

Luckily, with some AWS magic we can lean on inexpensive serverless offerings to achieve high availability, immediate positive response to webhooks, and the capacity to store the requests for later processing. AWS offers API Gateway, an HTTP-based service meant for receiving HTTP requests and returning responses. It’s scaleable, resilient, geographically distributed, and you can even set staged rollouts. AWS also offers Simple Queue Service (SQS), an extremely reliable service for delayed processing that Red Canary uses in the rest of our  ingestion pipeline. SQS is often paired with Simple Notification Service (SNS), which can “fan out” messages to multiple SQS queues.

There are many ways to use AWS’s API Gateway in a web stack, but its primary purpose is to provide a singular, highly available HTTP endpoint that can relay requests to a processor and then respond to the original request. It pairs well with AWS’s Lambda service, which can perform a small unit of processing. For more complicated features, it can route to arbitrary systems as long as they respond with some  form of HTTP.  Lucky for us, all SNS messages are submitted with an HTTP request, and we can get API Gateway and SNS to communicate directly, without involving Lambda or our own services.

AWS API Gateway webhooks processing flow

To meet our processing goals, we send every HTTP request that comes into our API Gateway straight into SNS, which will in turn deliver it to an SQS queue. The API Gateway will respond successfully to the webhook and at our leisure, we can process the messages from SQS complete with retries and scalable processing. It’s a completely asynchronous request-processing pipeline.

Your turn

The key is to make sure that the request that API gateway submits to SNS is in a format that SNS expects. When defining an integration request, API Gateway offers mapping templates to rewrite the incoming webhook body into a new SNS message. Here’s how to set it up.:

1. Configure an API Gateway method and select SNS as the AWS Service. Use POST as the HTTP method and set the content-type header to application/x-www-form-urlencoded
2. Specify an API Gateway mapping template. Use this example to take the webhook body and turn it into the SNS message body for the example SNS topic: