Burhan Deniz Abdi

Intro

So I've been using Sumologic for a while now. Its a great way to get up and running with centralised log monitoring very quickly. No need to install and maintain an ELK cluster. Just sign up and start throwing it logs and configure your dashboards!

Lately I had a need to integrate Sumologic's searching capabilities with a centralised alerting console I use. Here I am going to cover how I went about doing this with Sumologic and AWS. Note that the solution I am using here is not just applicable to Sumologic. Any system that has support for webhooks can use this solution.

Prerequisites

Before continuing with the rest of this article. Assumptions below have been made:

• You have an AWS account
• You have a Sumologic account
• Familiarity with Sumologic, AWS API Gateway, AWS Lambda, AWS SNS, AWS SQS and Boto3

Architecture

To achieve this I am utilising scheduled views in Sumologic to raise an alert based on my search and threshold, then using Webhooks to shoot off to an AWS API Gateway method backed by a Lambda function. This Lambda function will publishes to an SNS which in turn sends the message to SQS ready to get picked up by my alerting console. Below diagram describes this.

For the purpose of this example I will take us all the way to the SQS level. You may not want to go that far depending on your use case.

SNS topic and SQS

Starting from the end of the diagram we will create a standard SQS queue using the quick-create queue.

You should end up with a queue like below.

Then create your SNS topic and subscribe your SQS to it.

Lambda

Now we will take a look at the Lambda function powering the backend. See the code below:

import json
import boto3

sns = boto3.client('sns')

def lambda_handler(event, context):
    snsARN = '<ARN_OF_YOUR_SNS'
    data = event
    data["default"] = json.dumps(event)

    try:
        print("Publishing message to sns")
        sns.publish(TargetArn=snsARN, Message=json.dumps(data), MessageStructure='json', Subject="Alert")
        print("Published to sns")

    except Exception as e:
        print(e)
        raise e

This simple lambda is using boto3 to publish the webhook into an sns topic, which has our SQS subcribed to.

Important that you give this lambda function the correct permissions. To do this I have create a role in IAM with the following permissions and attached this role to my function:

API Gateway

The front end to this service is the AWS API Gateway. Firstly create your API by clicking the "Create API button" in the API Gateway console.
The API for simplicity and extensibility is very minimal by design. The below table describes the endpoint available:

POST /Sumo2Siren

This POST method provides the means for the Webhook from Sumologic to pass data in. Lets take a look at this method.

Nothing special here, you can set up your endpoint as you desire, I have opted to use API keys without any authorization. I will be using usage plans to specify connection limitations, we will see this soon.
Now lets take a look at the integration setup for your method:

Here I am using simple Lambda integration. The proxy integration is handy if you want the entire request object to get wrapped inside a JSON blob. In this use case the simple body of the request is enough.

You should end up with a workflow that looks like this:

API Keys and usage plan

Usage plans allow you to specify lower levels of control around what can be done with your API. You can enforce throttling, max hits, bursts and rate limits. If you haven't used Usage plans yet and don't see the option in your API console, you most likely need to enable it on your account.
In the Usage plan menu, create your plan. Like below:

Hit next when done, then skip on the next page as we still need to create our key.

Next create your API key. Go to the API Gateway console and select API keys, then create key. Once created click back on your key and you will see there is an "Add to Usage plan" option available. Hit that and you can now tie your key to the usage plan, like below:

Take a note of your API key by clicking on the "Show" button next to "API key"

Deploy API

Now you are ready to deploy your API. To do this you need to Stage it. Go to your API and click on "stage" menu option. Create a stage like below:

I advise you pick an appropriate stage name as this will make up part of your endpoint. I like using name like "vN" where N is the version iteration, as this allows for easy tracking of deployments. Once created you will get an "Invole URL" which is just an endpoint to your API, take note of this.

Sumologic Webhook

First you need to set up your Webhook connection in Sumologic. Log in and navigate to Manage --> Data Configuration --> Connections. Create a connection, in the wizard choose "Webhook":

Don't forget to place your URL in there, this will be the endpoint you noted earlier.
This payload comprises the body of the request, this is what will be passed to your Lambda function. For further payload examples see here.

Now that you have created your connection to your API, you need to set a scheduled view that will make use of this webhook once alerted.

Scheduled View

You can follow the instructions here. The last part of the puzzle is to connect your scheduled view to the webhook, this is done through the "connection" field:

Now sit back and watch those alerts fire when matching your condition.