ADF Run Pipeline

Description

This job triggers and monitors the running of a Pipeline in ADF (not a Pipeline in Orchestra)

Use Cases

We recommend creating a ADF Run Pipeline Task for every "Pipeline" within ADF.

This way, you can use Orchestra to trigger your reverse ELT on a cron or event based schedule. This has a number of advantages vs. using ADF's in-built scheduler:

• You can co-ordinate tasks outside of ADF- these would typically be other Spark jobs, other notebooks, or other tasks in ADF-adjacent environments e.g. Spark notebooks, python jobs

  • A common use-case is to have a dbt run jobs that run after ADF workflows that are running autoloader scrips, or Coalesce.io jobs

• You can use Orchestra to trigger jobs across ADF Accounts / Environments

• When ADF jobs run, Data Warehouse cost is incurred. Running these operations on a schedule you set explicitly ensures these costs do not go out of hand

• We aggregate metadata from the ADF Task in the same place as the metadata from other operations in your Pipeline

Parameters and setup

These parameters are required to run the Run Workflow Task

Name	Data type	Restrictions	Example
Resource group	String	N.A.	company-resource-group
Data factory	String	N.A.	company-data-factory
Pipeline name	String	N.A.	order-pipeline
Parameters	JSON	JSON format	{ "arg": "dummy" }
Subscription Id	String	Optional	UUID

Setup guide

Fetch the Azure Data Factory parameters

1. Head to your Azure Data Factory portal

2. Navigate to pipelines. From here you can get your Resource Group, Data Factory and Pipeline Name

We recommend leaving your pipelines without a trigger. This way Orchestra can manually trigger them when needed.

Options

You can send parameters to Azure pipeline run by copying a JSON object with your parameters.

You can optionally set the subscription ID, if left blank the value defaults to the value set for the selected credential.

API Requests

TBU