Run Workflow

Description

Triggers and monitors a GitHub Run Workflow job using the Create a workflow dispatch event endpoint.

Use Cases

There are a number of advantages to triggering workflow runs in Orchestra.

• Make use of Orchestra's cron scheduler

• Trigger and monitor multiple GitHub Workflow runs in a single DAG

• Trigger GitHub Workflow runs alongside other services such as Fivetran or Airbyte

• Gain end-to-end lineage across GitHub Workflow runs and upstream tasks

Parameters

Orchestra uses the Create a workflow dispatch event endpoint to trigger a workflow run. This endpoint allows you to trigger a workflow run for a specific workflow file in a repository.

The Create a workflow dispatch event endpoint requires the following parameters:

Parameters	Data type	Restrictions	Example
Owner	string	n/a	orchestra-hq
Repository	string	n/a	examples
Workflow ID	string	n/a	run.yaml
Ref	string	n/a	main
Input parameters	JSON	n/a

Run dbt core in GitHub Actions

Orchestra allows the Task to be configured for running dbt core in GitHub Actions. Orchestra will parse the artifacts generated from the Task to provide the lineage of the dbt run, and generate your data assets for you in the Orchestra platform.

To configure this Task for running dbt core in GitHub Actions:

• Select Collect additional metadata on the Task

• Select source integration as dbt Core

• (optional) Add the warehouse identifier to assist with generating more accurate data assets

In your GitHub Action, you will also need to upload artifacts to GitHub for Orchestra to download once the Task completes. An example GitHub Actions YAML file is shown below:

name: dbt core

on: workflow_dispatch

jobs:
  run-build-dbt:
    name: Run
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.12.x"
      - name: Full Deploy Commands
        working-directory: ./dbt_on_gha
        run: |
          python3 -m venv venv
          source venv/bin/activate
          venv/bin/python3 -m pip install --upgrade pip
          python3 -m pip install -r requirements.txt
          source venv/bin/activate
          dbt debug
          dbt build
        shell: bash
      - name: Upload manifest.json to GitHub
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: manifest.json
          path: ./dbt_on_gha/target/manifest.json
          retention-days: 1
          overwrite: true
      - name: Upload run_results.json to GitHub
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: run_results.json
          path: ./dbt_on_gha/target/run_results.json
          retention-days: 1
          overwrite: true

Notes:

• The uploading of the artifacts must happen after dbt has executed, and if: always() is included to ensure we collect lineage data even if part of the dbt command fails

• We recommend using the actions/upload-artifact@v4 action provided by GitHub

• The name property must be set as manifest.json and run_results.json

  • If you have multiple run result files, they can be uploaded with a suffix such as _1 and _2. For example, run_results_1.json and run_results_2.json. This will help Orchestra build an accurate lineage view. This is only necessary for multi dbt step processes