Datadog Integration

Create checks for Datadog monitors and alerts.

Add a Datadog Integration

  1. In the OpsLevel app, Click Integrations in the left sidebar.
  2. Click on the + New Integration button.
  3. Click the Datadog tile to add the integration.

Create a Check

  1. Navigate to the Rubrics sub menu under the Service Health menu in OpsLevel
  2. Hover over the cell that corresponds to the level and category you want your check to live in and click the + Add Check button. Add Check
  3. Create a Custom Event check
  4. Select the Datadog integration you created from the Integration dropdown
  5. Choose one of the Check Templates from the dropdown.

    Check Template Here is what each template is for:

    • Check For SLO Error Budget Alert: The check will fail if the alert is triggered.
  6. We will populate the Service Specifier field, which we use to determine what service to run the check for, and the Success Condition field, which we use to determine if the check should pass or fail. We also provide a sample payload to test the check. Service Specifier and Success Condition

Send payload to OpsLevel

For any Datadog metric alert, you can create a Webhook using the URL from the integration you created above. See here for more details.

Webhook URL

If the payload you will be sending in the webhook does not have a service identifier present, we recommend appending a query parameter to your URL like the following: ?serviceAlias=my-service-alias

The minimum set of payload variables we recommend are:

{
    "id": "$ID",
    "body": "$EVENT_MSG",
    "alertId": "$ALERT_ID",
    "eventMessage": "$TEXT_ONLY_MSG",
    "title": "$EVENT_TITLE",
    "url": "$LINK",
    "alertTransition": "$ALERT_TRANSITION",
    "hostname": "$HOSTNAME",
    "orgName":"$ORG_NAME",
    "priority":"$PRIORITY",
    "snapshot": "$SNAPSHOT",
    "alertQuery": "$ALERT_QUERY",
    "alertScope": "$ALERT_SCOPE",
    "alertStatus": "$ALERT_STATUS",
    "eventType": "$EVENT_TYPE",
    "lastUpdated":"$LAST_UPDATED"
}

To use your webhook, add @webhook-<WEBHOOK_NAME> in the Notify your team section when creating or editing a monitor alert.

Webhook Notification Example