⏰Alerting

Avoid alert fatigue with custom-alerts

One of the biggest problems data engineering teams face is alert-fatigue. There is a fine balance between understanding 100% of pipeline activity and having too many alerts. Orchestra has a number of features that help data teams avoid alert fatigue.

Set-up

To set up alerting for a Pipeline, Head over to the Pipelines Page and "Edit" by the side menu

In the Pipeline Builder, hit the menu icon in the top right hand side.

This prompts you to enter a few pieces of information

  1. Alert name: a unique name for your alert. e.g. "Data-Engineering Failures"

  2. A resource: you can configure alerts on a whole Pipeline or a Task

  3. Alert on Status: when Tasks and Pipelines finish, they end up in a state like Completed, Failed, Skipped, Cancelled. You can configure alerts on specific states

The final section is destinations.

You can send an alert to one or multiple destinations. This is powerful for triaging different alerts with different severities into different places.

Orchestra supports granular alerting which avoids alert fatigue:

Alert message structure

Using Orchestra has a number of advantages in terms of alerting due to its ability to consolidate information into a single place. Rather than understand alerts from multiple providers, Orchestra provides Data teams with a uniform and helpful error format.

There are a few points to note:

  • The hyperlink brings the user to the Task Runs page in Orchestra

  • The failed state corresponds to the state the Task finished in

  • ⚙️ Is the name of the Pipeline

  • 📜 Is the Orchestra message - a succinct description of the cause of failure

  • 🎯 Is the name of the Integration Job

Integrations

  • Slack

  • Teams (Beta)

  • Email (Beta)

  • In-app

Slack

Once you have connected Orchestra to Slack from the Connectors page, Orchestra will be able to view all public channels in your Slack workspace. To allow Orchestra to send messages to private channels you must add the Orchestra App to that channel. This is done in Slack by typing /add and then selecting the Orchestra App in the popup.

Additional configurability (Alpha)

We have a number of features currently in Alpha that will allow pipelines and tasks to have configurable alert settings based on severity. A common implementation pattern we recommend is:

  • Decide on how many degrees of severity you wish to have e.g. P1, P2, P3

  • Decide which Pipelines and Tasks' failures should correspond to what severity of error

  • For each error severity, set up an alerting channel e.g. a separate email distribution list or Slack channel

  • Configure the alerts by Pipeline / Task depending on their severity

Future features

  • Configure alerts for multiple channels

  • Configure alerts for email addresses

  • Severity

  • Task-level alerting

    • Get an alert any time something happens to a task

PreviousSecurityNextArtifacts

Last updated