Custom Event Checks

Send arbitrary JSON payloads to OpsLevel to evaluate the maturity of your services. Using jq, you can parse out service identifiers from your payload and determine whether the check, scoped to each found service, passes or fails. You can also customize check result messages using Liquid templates.

OpsLevel helps keep your services consistent and healthy through the use of various pre-built checks. However, not all checks fall in line with one of the pre-built checks provided by OpsLevel.

With custom event checks, you can easily integrate with any third party platform to create custom, data driven checks. The result message has the full context which can be templated to provide service owners with an informative message.

Custom event checks allow you to evaluate various conditions by sending any JSON payload to OpsLevel. The following are examples of conditions you can evaluate using these checks:

  • Using your vulnerability detection tool to ensure that no Tier-1 services have open vulnerabilities.
  • Using your vulnerability detection tool to ensure no Ruby services have high criticality vulnerabilities.
  • Ensuring your test coverage stays above a certain threshold.

Getting Started with Custom Event Checks

Step 1: Create a Custom Event Integration

To set up a custom event check endpoint, visit the Integrations tab and select “New Integration”, followed by Custom Event.

Integrations page

Click + Add Integration on the Custom Event Integration card, enter the name of the system you are integrating with (e.g. PagerDuty) and click “Create”.

Integration page with arrow to the custom event integration

After creating a Custom Event Integration, you’ll be redirected to a page that looks like:

Custom event page

Step 2: Create a Custom Event Check

To create a custom event check, navigate to the Rubrics tab

Either in the top right corner, or inside your rubric, press the + Add Check button.

Adding a check

From the Create Check modal, select Custom Event Check from the Type dropdown:

Selecting a check type

Select the Integration you previously created, it’s URL will be displayed below the dropdown. This is where you should be sending your JSON payloads for the check. Selecting the integration

To identify your service, or services, in the sent JSON payload, you can specify a Service Identifier jq expression that will find your service aliases in the payload or query params. Refer to the service identifier section for more information and examples.

To evaluate the success of your check, specify a Success Condition jq expression that will be run against your payload to extract a boolean value to determine whether your check is passing or failing. Refer to the success condition section for more information and examples.

You can enter a Result Message to provide feedback to users regarding the success or failure of your check. The message template supports Markdown and Liquid templating. For more information and examples, refer to the Liquid Templating section.

Adding result messages

In the Test Check section, you can test your check before creating it. Enter a sample JSON payload that resembles what you’ll be sending to the custom check endpoint and your inputs will be used to evaluate your expression, display the found services, and render your Result Message.

Note: Only the check for the first found service in your payload will be evaluated for the test.

Testing check

After successfully creating a check, you will see it in your rubric.

Step 3: Sending a JSON Payload to OpsLevel

Open one of your Custom Event Checks in your rubric, and copy the URL present in the modal. This is the endpoint you’ll be sending your JSON payloads to.

Using that endpoint, make an HTTP POST request like the example below, substituting your custom check endpoint URL in place of  xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx:

curl -X POST https://app.opslevel.com/integrations/custom_event/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx \
-H 'content-type: application/json' \
-d '
{
  "messages": [
    {
      "event": "incident.trigger",
      "incident": {
        "incident_key": "12",
        "status": "resolved",
        "service": {
          "id": "service1",
          "name": "Production XDB Cluster",
          "description": "This service was created during onboarding on July 5, 2017.",
          "auto_resolve_timeout": 14400
        }
      },
      "id": "69ad-11e7-a799-22000a15839c7ced0-a2c",
      "created_on": "2017-09-26T15:14:36Z"
    },
    {
      "event": "incident.trigger",
      "incident": {
        "incident_number": 33,
        "status": "resolved",
        "service": {
          "id": "service2",
          "name": "Production XDB Cluster",
          "description": "This service was created during onboarding on July 5, 2017.",
          "auto_resolve_timeout": 14400,
          "acknowledgement_timeout": 1800,
          "created_at": "2017-07-05T17:33:09Z",
          "status": "critical",
          "last_incident_timestamp": "2017-09-26T15:14:36Z"
        },
        "id": "69a7ced0-99-22000a15839ca2cd-11e7-a7",
        "created_on": "2017-09-26T15:14:36Z"
      }
    }
  ]
}'

If your request was successful, you will receive the following response:

{ "result": "ok" }

Examples

Single Service Payload: Check for Vulnerabilites in yarn.lock using Snyk

We can use Snyk to find security vulnerabilities in our projects. Specifically we’ll assess our yarn.lock file and send the results to OpsLevel.

snyk test --prune-repeated-subdependencies --json --file=yarn.lock | curl -X POST http://app.opslevel.com/integrations/custom_event/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx -H 'content-type: application/json' --data-binary @-

We use the Snyk CLI to assess our yarn.lock file and pipe the JSON output to a curl command which POSTs to our custom event endpoint.

Note: We use the --prune-repeated-subdependencies flag to compress our Snyk payload due to our 2MB payload limit.

The payload sent by Snyk will look something like this:

{
  "vulnerabilities": [
      {
        "severity": "medium",
        "severityWithCritical": "medium",
        "title": "Regular Expression Denial of Service (ReDoS)",
        "from": [
          "app"
        ],
        "upgradePath": [],
        "isUpgradable": false,
        "isPatchable": false,
        "isPinnable": false,
        "name": "lorem ipsum",
        "version": "4.5.394"
      },
      {
        "severity": "high",
        "severityWithCritical": "high",
        "title": "Regular Expression Denial of Service (ReDoS)",
        "from": [
          "app"
        ],
        "upgradePath": [
          false
        ],
        "isUpgradable": true,
        "isPatchable": false,
        "isPinnable": false,
        "name": "lorem ipsum",
        "version": "6.0.1"
      }
  ],
  "filesystemPolicy": false,
  "filtered": {
    "ignore": [],
    "patch": []
  },
  "uniqueCount": 2,
  "projectName": "app_name",
  "foundProjectCount": 1,
  "displayTargetFile": "yarn.lock",
  "path": "/path/to/project"
}

For our check, we’ll write a Service Identifier to parse the service alias from the JSON payload.

.projectName

We’ve identified our service, now we can write our Success Condition for the payload.

([.vulnerabilities[] | select(.severity == "high")] | length == 0) and ([.vulnerabilities[] | select(.severity == "medium")] | length <= 3)

Here we’re searching inside the vulnerabilites array for items with high and medium severities, and checking if there are three or less.

Finally, we’ll craft our result message using a combination of Markdown and Liquid templating.

{% if check.passed %}
  ### Check passed
{% else %}
  ### Check failed

  service **{{ data.projectName }}** has **{{ data.vulnerabilities | where: "severity", "high" | size }}** high vulnerabilities and **{{ data.vulnerabilities | where: "severity", "medium" | size }}** medium vulnerabilities.

It has **{{ data.uniqueCount }}** vulnerabilities in total.
{% endif %}

Multi-Service Payload: Check for PagerDuty Triggered Incidents

In PagerDuty, you can specify a webhook endpoint to send events that happen in your account, this would be the URL to your custom check integration.

We’ll write a Service Identifier to parse your service aliases from the JSON payload.

.messages[] | .incident.service.id

Now that we have our services identified we can work on writing a Success Condition for the payload, scoped down to the section relevant to each found service.

.messages[] | select(.incident.service.id == $ctx.alias) | .incident.status == "resolved"

Here we are finding each service in the list of messages, and passing the check if the incident has been resolved. The other possible values, acknowledged or triggered, will fail the check.

Finally, we’ll craft our result message using a combination of Markdown and Liquid templating.

{% if check.passed %}
  ### Check passed
{% else %}
  ### Check failed
  **{{ data.messages[ctx.index].incident.service.id }}** has an unresolved incident.
{% endif %}

Using URL Query Params: Check for Number of Issues in Sentry

In Sentry you are able to specify a webhook to receive event payloads. To identify the service, we will specify it using query params.

In Sentry, navigate to your project and select Settings > Integrations, then search for “Webhook”. Select it and add it to the project of interest.

Finding Sentry Webhooks Adding To Project

In your Sentry project you can configure your Sentry project to send webhooks. In this example we will configure it to send payloads to the custom event integration.

Sentry Webhook Configuration

Once this is configured Sentry will begin sending webhooks to OpsLevel. For our check we will use the $params variable in our service specifier.

$params.alias

A completed check will look like:

Example Sentry Check in OpsLevel

Custom Check Fields

Service Identifier

When crafting service identifier expression, OpsLevel provides access to your URL query params to help route your check in the scenario where your payload does not include a service alias.

Variable Description
$params The URL query parameters provided in the POST request sent to the Custom Event Integration endpoint.

Success Condition

When crafting your success condition, OpsLevel provides access to variables to help specify which part of the payload is relevant to each found service.

Variable Description
$ctx The context object for each found service alias.
$ctx.alias The service alias found in your payload.
$ctx.index The index in the list where the service was found.
$ctx.count The number of services found.
$params The URL query parameters provided in the POST request sent to the Custom Event Integration endpoint.

Another way of looking at it is that $ctx is an object:

{
  "alias": "billing",
  "index": 2,
  "count": 5
}

Liquid Templating

When creating custom result messages with Liquid, OpsLevel provides access to variables to help in crafting helpful messages.

Note that the variables in liquid templating do not include a dollar sign ($) at the front.

Variable Description
check The check object contains information about the check result
check.passed A boolean value of whether the check passed.
check.failed A boolean value of whether the check failed.
check.status A string of the current check status, passed or failed.
data The data object gives you access to the payload sent to the OpsLevel Custom Event Integration endpoint.
ctx The context object for each found service alias that contains alias, index, and count — just as in the Success Condition
params The URL query parameters parameters provided in the POST request sent to the Custom Event Integration endpoint.

Liquid Filters

Filters are simple expressions you can use to transform text and variables. For example, to generate the string filtrd txt, you can write:

{{ "Filtered TEXT" | downcase | remove: "e" }}

In addition to the default Liquid filters, you can use the jsonify filter to show data in the JSON format. See the example below.

Payload Check jsonify Filter

Example: Display the Number of High Vulnerabilities on your Service’s Repositories

In this example, a user wants to evaluate if there are any high criticality vulnerabilities on their service’s repo. They can send a simple JSON payload to OpsLevel:

{
  "vulns": {
    "high": 0
  }
}

If there are any high criticality vulnerabilities, we want to show the number of such vulns. Otherwise, if there are zero vulns, we want to show a “Congratulations” message.

The result message template looks like this:

{% if check.passed %}
## Congratulations!
{% else %}
## Critical Vulnerabilities Detected
You have {{ data.vulns.high }} high vulnerabilities.  
{% endif %}

Pass Result Message

Simple Liquid Template Pass Result

Fail Result Message

Simple Liquid Template Fail Result

Example: Display Detailed Vulnerability Information about your Service’s Repositories with Snyk

In this example, a user again wants to evaluate if there are any high criticality vulnerabilities on their service’s repo. However, in this case, the JSON payload contains additional detail about each vulnerability that can be shown on the check result message.

Using the Snyk API, a user can get detailed information about each vulnerability and send it to OpsLevel:

{
  "results": [
    {
      "issue": {
        "title": "Denial of Service (DoS)",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-1071903",
        "severity": "high"
      }
    },
    {
      "issue": {
        "title": "Cross-site Scripting (XSS)",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-536099",
        "severity": "medium"
      }
    },
    {
      "issue": {
        "title": "SQL Injection",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-472693",
        "severity": "medium"
      }
    }
  ]
}

The user can add the following jq expression to check if there are any high vulnerabilities:

[ .results | .[] | select(.issue.severity == "high") ] | length == 0

Then the user can use the following Liquid template to show a result message with information about how to resolve each vulnerability:

{%- if check.status == 'passed' %}  
  **Congrats**. No high criticality vulns found.
  {%- assign medium_vulns = data.results | map: "issue" | where: "severity", "medium" %}
  {%- if medium_vulns.size > 0 %}
    Want to be awesome? Fix these medium ones:
    {%- for vuln in medium_vulns %}
- [{{vuln.title}}]({{vuln.url}})
    {%- endfor %}
  {%- endif %}
{%- else %}
  ### Action Required: You have critical security vulns.
  Please fix the following immediately:
  {%- assign high_vulns = data.results | map: "issue" | where: "severity", "high" %}
  {%- for vuln in high_vulns %}
- [{{vuln.title}}]({{vuln.url}})
  {%- endfor %}
{% endif %}

Pass Result Message

Advanced Liquid Template Pass Result

Fail Result Message

Advanced Liquid Template Fail Result

Liquid Error Messages

When authoring templates in Liquid, you may experience one of the following Liquid related error messages.

Title Message Description
Liquid Error undefined variable One of the variables used in the Liquid template is not available. This can happen if the variable is not sent as part of the JSON payload (e.g., You refer to {{ data.does_not_exist }}).
Liquid Error undefined filter One of the filters used in the Liquid template is not available. See the filters section above for info on available filters.
Liquid syntax error varies depending on the error that occurred The Liquid template provided is syntactically invalid. Please refer to the official Liquid documentation to try and resolve this error.