⏰Alerting

Avoid alert fatigue with custom-alerts

One of the biggest problems data engineering teams face is alert-fatigue. There is a fine balance between understanding 100% of pipeline activity and having too many alerts. Orchestra has a number of features that help data teams avoid alert fatigue.

Set-up

To set up alerting for a Pipeline, head over to the Pipelines Page and click the edit button.

Hover over a pipeline to reveal the button to edit a pipeline

In the Pipeline Builder, hit the menu icon in the top right hand side.

This prompts you to enter a few pieces of information

1. Alert name: a unique name for your alert. e.g. "Data-Engineering Failures"

2. A resource: you can configure alerts on a whole Pipeline or a Task

3. Alert on Status: when Tasks and Pipelines finish, they end up in a state like Completed, Failed, Skipped, Cancelled. You can configure alerts on specific states

4. Custom message: optional message to add to the alert

The final section is destinations.

We support Slack, Teams and Email

You can send an alert to one or multiple destinations. This is powerful for triaging different alerts with different severities into different places.

Orchestra supports granular alerting which avoids alert fatigue:

Alert message structure

Using Orchestra has a number of advantages in terms of alerting due to its ability to consolidate information into a single place. Rather than understand alerts from multiple providers, Orchestra provides Data teams with a uniform and helpful error format.

A typical set of error alerts

There are a few points to note:

• The hyperlink brings the user to the Task Runs page in Orchestra

• The failed state corresponds to the state the Task finished in

• ⚙️ Is the name of the Pipeline
• 📜 Is the Orchestra message - a succinct description of the cause of failure
• 🎯 Is the name of the Integration Job

Integrations

• Slack

• Teams

• Email

• In-app

Slack

Once you have connected Orchestra to Slack from the Connectors page, Orchestra will be able to view all public channels in your Slack workspace. To allow Orchestra to send messages to private channels you must add the Orchestra App to that channel. This is done in Slack by typing /add and then selecting the Orchestra App in the popup.

Mentions

We support tagging a user in a Slack alert message. You can do this by adding a tag to the alert custom message <@U012AB3CD> where U012AB3CD is the member ID for the user you want to tag.

You can manually retrieve a specific user's ID by clicking the overflow button in their Slack profile, and choosing the Copy member ID option.

We do not currently support mentions for Teams alerts.

Additional configurability (Alpha)

We have a number of features currently in Alpha that will allow pipelines and tasks to have configurable alert settings based on severity. A common implementation pattern we recommend is:

• Decide on how many degrees of severity you wish to have e.g. P1, P2, P3
• Decide which Pipelines and Tasks' failures should correspond to what severity of error

• For each error severity, set up an alerting channel e.g. a separate email distribution list or Slack channel

• Configure the alerts by Pipeline / Task depending on their severity

Future features

• Severity