Execute Pipeline

Description

This job triggers a pipeline execution using this endpoint.

Use Cases

We recommend creating an Execute Pipeline Task for every "published Pipeline" within Matillion DPC.

This way, you can use Orchestra to trigger your ELT on a manual or event based schedule. This has a number of advantages vs. using Matillion's in-built scheduler:

• You can co-ordinate tasks outside of Matillion DPC, like other Matillion DPC jobs or tasks in other pieces of infrastructure like AWS EC2 using Orchestra

• When Matillion Pipelines run, Data Warehouse cost is incurred. Running these operations on a schedule you set explicitly ensures these costs do not go out of hand

• We aggregate metadata from the Matillion DPC Task in the same place as the metadata from other operations in your Pipeline

Parameters

These parameters are required to run the Execute Pipeline Task

Name	Data type	Restrictions	Example
Project ID	String	N.A.	27505728-3e7d-4dfd-*******************
Pipeline name	String	N.A.	first_pipeline
Environment name	String	N.A.	Production

Fetch the variables

To execute a pipeline in Matillion DPC, you need to fetch the project id, pipeline name and the environment name. Although these are arbitrary user-set strings, these act as the UIDs in Matillion DPC. You can find these ids in the Matillion DPC UI or using the Matillion DPC endpoints.

You can find the project id in the URL when you have loaded the pipeline you wish to run. The pipeline name and environment name should be visible from the pipeline editor page as well.

Alternatively you can follow this guide to extract the values you need via the API.

FYIs and Gotchas

ELT Pipelines

Matillion DPC offers a Fivetran-style UI for quickly ingesting data from source to target. This is not recommended by Orchestra because these jobs cannot be co-ordinated via API. Fortunately, anything that can be run as an "ELT Pipeline" can also be run as a regular "Published Pipeline" in Matillion DPC, so we recommend pushing all logic into a Matillion "Published Pipeline".

We do not recommend creating ELT pipelines using the UI above

Published Pipelines

Due to the Git-based version control system inherent in Matillion, it's impossible to run a Pipeline before it is "Published" - i.e. commited to Git. In order to do this, ensure you head over the the Hub and select a Project before starting on a Pipeline. Once the Pipeline is ready to be executed, you should head over to the "Git" tab (in the top right-hand corner) and commit changes. You should then push local changes, ensuring the "Publish Pipelines" Toggle is switched on (see below).

Ensure Publish Pipelines is selected

The Matillion UI used for Publishing Pipelines can be seen below.

Error handling

API Requests

If we receive the following error codes from Matillion, we'll raise an error and the task will move to a failed state.

Code	Description	Handling
401	Unauthorised	We will raise an error and parse the raw error message from the Matillion response as the Orchestra message
404	Not Found	We will raise an error with the HTTP Reason as the Orchestra message
Other error code		We will raise an error with the HTTP Reason as the Orchestra message