Run Query

Runs a query in BigQuery

Description

This job asynchronously runs a query in BigQuery. Documentation can be found here.

Use Cases

We recommend creating a BigQuery Run Query Task for every "Query" you need to run within BigQuery.

In BigQuery, pretty much everything you can do can be done using a query. This ranges from simple SQL statements to anything within the DDL/DML spectrum. Furthermore, you can also run Data Quality tests and clone tables using BigQuery Queries.

There are a number of advantages to triggering BigQuery queries in Orchestra vs. other workflow orchestration tools

  • You can co-ordinate tasks outside of BigQuery - these would typically be other BigQuery jobs, data loading steps, or other tasks in BigQuery-adjacent environments e.g. dbt, a BI tool

  • You can use Orchestra to trigger jobs across BigQuery Projects / Environments

  • When BigQuery jobs run, Data Warehouse cost is incurred. Running these operations on a schedule you set explicitly ensures these costs do not go out of hand

  • We aggregate metadata from the BigQuery Task in the same place as the metadata from other operations in your Pipeline

Parameters

These parameters are required to run the BigQuery Run Query Task

NameData typeRestrictionsExample

Query

String

N.A.

SELECT * FROM TABLE

Location

String

Optional (overrides value used in credential object)

europe-west2

PreviousGCP - BigQueryNextDatabricks

Last updated