Payload Checks

Deprecation Note: It’s recommended to use the newer, more flexible Custom Event Check.

OpsLevel helps keep your services consistent and mature through the use of various pre-built checks. However, not all checks fall perfectly in line with one of the pre-built checks provided by OpsLevel.

With payload checks, you can easily create custom, data driven checks and format the result messages to provide the context you want.

These checks allow you to evaluate various conditions by sending any JSON payload directly to OpsLevel. The following are examples of conditions you can easily evaluate using these checks:

Using your vulnerability detection tool to ensure that no Tier-1 services have open vulnerabilities.
Using your vulnerability detection tool to ensure no Ruby services have high criticality vulnerabilities.
Ensuring your test coverage stays above a certain threshold.

You could implement examples like the above with Custom Checks, but Payload Checks make specifying thresholds significantly easier.

Getting Started with Payload Checks

Step 1: Create a Payload Check Integration

To set up a payload check endpoint, visit the Integrations tab and select “New Integration”, followed by Payload.

OpsLevel New Integration

Click + Add Integration on the Payload Integration card and then New Payload Integration:

OpsLevel Payload Integration

After creating a Payload Integration, you’ll be redirected to a page that looks like:

Payload Integration Show

Step 2: Create a Payload Check

Payload checks are a type of check that allow you to send arbitrary JSON payloads to an OpsLevel endpoint. You can evaluate that payload against a jq expression and return a custom check result message using Liquid templating.

To create a payload check, navigate to the Checklists tab.

OpsLevel Checks Tab

If you already have checklists created, you will see them here. If not, visit the Checklists page to learn more about checks.

For the purpose of this guide, we will be using our All Services checklist.

Navigate to the checklist you wish to create a check for.

OpsLevel Checklists

Press the + Add Check button.

OpsLevel Add Check

From the Create Check modal, select Payload Check from the Type dropdown:

OpsLevel Create Check Modal

Fill out the check information and press Create.

You can enter a jq Expression which will be run against the input check payload to determine if the check is passing or failing. Refer to the jq Expressions section for more information and some examples.

You can enter a Result Message to provide a custom check result message which supports both Markdown and Liquid templating. For more information and some examples, refer to the Liquid Templating section.

OpsLevel Payload Check Information

In the Test Check section, you can test out your check before releasing it into the wild. You can enter a sample JSON payload that the jq Expression will be evaluated against. Then the Result Message will be evaluated and you can see the final result in the Test Result.

Payload Check Test Result

After successfully creating a check, you will see it in the Checks card.

OpsLevel Check Created

From here you can grab the Payload Check Identifier.

Sending a JSON Payload to OpsLevel

It is easy to start sending payloads for a payload check to a particular service in OpsLevel. Begin by navigating to the check result by visiting the Scorecards tab of your service. Hint: The check result will be yellow if you haven’t send a payload to that check for the specific service.

OpsLevel Check alias and Identifier

Using the Payload Check Identifier and any Service Alias provided, make an HTTP Post request like the example below, substituting your Webhook URL in place of xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx:

POST https://app.opslevel.com/integrations/payload/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx HTTP/1.1
Content-Type: application/json

{
  "service": "shopping_cart",
  "check": "checkIdentifier",
  "data": {
    "_comment": "YOUR DATA HERE"
  }
}

If your webhook was successful, you will receive the following response:

{
  "result": "ok"
}

Here is a full explanation of all the fields in the json object:

Attribute	Type	Required	Description
service	String	true	The alias that corresponds to the service you wish to create a check result for. This can be found on the Summary tab for a service.
check	String	true	The check reference id that corresponds to the custom check you are creating a check result for. Refer to the Getting Started with Payload Checks section to see how you get this.
data	JSON	true	The JSON payload you wish to evaluate your check against.

Handling Errors

The Payload check API may throw errors when receiving malformed requests, when utilizing endpoints that are no longer valid, or when we’re too busy trying to determine who the werewolf is among us to route your webhooks correctly.

In such cases, payload webhooks return verbose and descriptive errors.

An example of such error:

{
  "errors": [
    {
      "status": 422,
      "title": "No service found with alias 'invalid_alias'.",
      "detail": null
    }
  ]
}

The error above notifies the user that even though we received the webhook payload, the service provided was not found in your OpsLevel account.

Common Errors

Error Code	Title	Description
422	No service found with alias ‘invalid_alias’.	The service alias provided does not exist in your OpsLevel account.
422	Check not found with id ‘invalid_id’.	The check id provided did not match any check on your account.
422	Expected a top level field named ‘data’ but none was found.	The check webhook payload requires a `data` field but none was provided.

jq Expressions

Example: Check for Vulnerabilities on Your Service’s Repositories with Snyk

When using Snyk, you may want to ensure that the repos used by your service have no high criticality security vulnerabilities. You can use Snyk’s API to fetch vulnerability counts.

Send a POST request to the following Snyk API Endpoint:

https://snyk.io/api/v1/reporting/counts/issues/latest?groupBy=severity

You will receive a response similar to:

{
  "results": [
    {
      "day": "2017-07-01",
      "count": 13,
      "severity": {
        "high": 0,
        "medium": 4,
        "low": 9
      }
    }
  ]
}

Forward this to OpsLevel’s payload check endpoint.

For a full example, do the following.

Get an API token and Organization ID from Snyk.

Set up the following environment variables:

SNYK_TOKEN=<Snyk API Token>
SNYK_ORGANIZATION=<Snyk Organization ID>
OPSLEVEL_SERVICE_ALIAS=<OpsLevel Service Alias>
OPSLEVEL_CHECK_ID=<OpsLevel Check ID>
OPSLEVEL_PAYLOAD_URL_TOKEN=<OpsLevel Payload URL Token>

Run the following command:

curl --X POST 'https://snyk.io/api/v1/reporting/counts/issues/latest?groupBy=severity' \
--H 'Content-Type: application/json' \
--H 'Authorization: '"$SNYK_TOKEN"'' \
--data-raw '{
 "filters": {
     "orgs": ["'"$SNYK_ORGANIZATION"'"]
 }
}' |\
curl -X POST 'http://app.opslevel.com/integrations/payload/${OPSLEVEL_PAYLOAD_URL_TOKEN}' \
-H 'content-type: application/json' \
-d '{
  "service": "'$OPSLEVEL_SERVICE_ALIAS'",
  "check": "'$OPSLEVEL_CHECK_ID'",
  "data": '$(</dev/stdin)'
}'

Problem	Expression	Result
Ensure you have no vulnerabilities on the most recent day.	`.results \| .[-1].count == 0`	This expression will return `false` and cause your `Payload Check` to fail because your repository has `13` vulnerabilities.
Ensure you have no high vulnerabilities.	`.results \| .[-1].severity \| .high == 0`	This expression will return `true` and result in a passing `Payload Check` since your repository has `0` high severity vulnerabilities.

Example: Check for Correct API Version on Kubernetes Deploy

In this example we will be using an example service called cartservice.

If you use Kubernetes and wish to check if you are using the correct API version on deploy, you can run the following command:

kubectl get deployment cartservice -o json

The resulting JSON payload can be forwarded to OpsLevel to run Payload Checks against. The payload will look similar to:

{
    "apiVersion": "apps/v1",
    "kind": "Deployment",
    # ...
    "status": {
        "availableReplicas": 1,
        # ...
    }
}

Problem	Expression	Result
Validate that the `apiVersion` for all services is `apps/v1`	`.apiVersion == "apps/v1"`	This expression will return `true` and result in a passing check because the `apiVersion` is indeed `apps/v1`.
Check that your deployment has 3 `availableReplicas`	`.status.availableReplicas == 3`	This expression will return false and result in a failing check because you only have 1 available replica.

jq Error Messages

When creating a payload check, you may experience one of the following jq related error messages.

Message	Description
jq expression cannot be empty.	You cannot provide an empty jq expression when creating a payload check.
The given jq expression is not syntactially valid.	There is a syntax error in the `jq` expression provided. Please refer to the official jq documentation to try and resolve this error.
jq expression could not be evaluated.	An evaluation error occurred with your `jq` expression. This is most likely caused by providing a payload that does not contain the necessary information to evaluate the `jq` expression against.
Content cannot be empty.	The sample JSON payload provided cannot be an empty object.

Liquid Templating

Payload checks allow you to use Markdown combined with Liquid templating to create your own custom result messages.

Liquid Template Variables

When creating custom result messages with Liquid, OpsLevel gives you access to some variables to use in the Liquid template.

Variable	Description
check	The `check` object contains information about the check result
check.passed	A boolean value of whether the check passed
check.failed	A boolean value of whether the check failed
check.status	A string of the current check status, `passed` or `failed`
data	The `data` object gives you access to the `data` portion of the payload you sent to the OpsLevel Payload Check endpoint.

Liquid Filters

Filters are simple expressions you can use to transform text and variables. For example, to generate the string filtrd txt, you can write:

{{ "Filtered TEXT" | downcase | remove: "e" }}

In addition to the default Liquid filters, you can use the jsonify filter to show data in the JSON format. See the example below.

Payload Check jsonify Filter

Example: Display the Number of High Vulnerabilities on your Service’s Repositories

In this example, a user wants to evaluate if there are any high criticality vulnerabilities on their service’s repo. They can send a simple JSON payload to OpsLevel:

{
  "vulns": {
    "high": 0
  }
}

If there are any high criticality vulnerabilities, we want to show the number of such vulns. Otherwise, if there are zero vulns, we want to show a “Congratulations” message.

The result message template looks like this:

{% if check.passed %}
## Congratulations!
{% else %}
## Critical Vulnerabilities Detected
You have {{ data.vulns.high }} high vulnerabilities.  
{% endif %}

Pass Result Message

Simple Liquid Template Pass Result

Fail Result Message

Simple Liquid Template Fail Result

Example: Display Detailed Vulnerability Information about your Service’s Repositories with Snyk

In this example, a user again wants to evaluate if there are any high criticality vulnerabilities on their service’s repo. However, in this case, the JSON payload contains additional detail about each vulnerability that can be shown on the check result message.

Using the Snyk API, a user can get detailed information about each vulnerability and send it to OpsLevel:

{
  "results": [
    {
      "issue": {
        "title": "Denial of Service (DoS)",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-1071903",
        "severity": "high"
      }
    },
    {
      "issue": {
        "title": "Cross-site Scripting (XSS)",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-536099",
        "severity": "medium"
      }
    },
    {
      "issue": {
        "title": "SQL Injection",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-472693",
        "severity": "medium"
      }
    }
  ]
}

The user can add the following jq expression to check if there are any high vulnerabilities:

[ .results | .[] | select(.issue.severity == "high") ] | length == 0

Then the user can use the following Liquid template to show a result message with information about how to resolve each vulnerability:

{%- if check.status == 'passed' %}  
  **Congrats**. No high criticality vulns found.
  {%- assign medium_vulns = data.results | map: "issue" | where: "severity", "medium" %}
  {%- if medium_vulns.size > 0 %}
    Want to be awesome? Fix these medium ones:
    {%- for vuln in medium_vulns %}
- [{{vuln.title}}]({{vuln.url}})
    {%- endfor %}
  {%- endif %}
{%- else %}
  ### Action Required: You have critical security vulns.
  Please fix the following immediately:
  {%- assign high_vulns = data.results | map: "issue" | where: "severity", "high" %}
  {%- for vuln in high_vulns %}
- [{{vuln.title}}]({{vuln.url}})
  {%- endfor %}
{% endif %}