GCP - BigQuery

Run queries in BigQuery

Type: Warehouse

Website: https://cloud.google.com/bigquery

General docs: https://cloud.google.com/bigquery/docs

API Docs: https://cloud.google.com/bigquery/docs/reference/rest

Authentication

To connect BigQuery to Orchestra, you will need:

• a GCP account with BigQuery enabled in at least one project

• the ability to create a service account in GCP, or create a new key for an existing service account

• a dataset in BigQuery

Instructions

Service Account JSON

Orchestra connects to your GCP account via a service account. More information available here.

We recommend creating a new service account for Orchestra to use that only has the ability to access BigQuery in a given project. You can create a new service account in the GCP console. If you reuse an existing service account, you will need to ensure that it has the correct permissions to access BigQuery.

• Navigate to Google Cloud IAM Admin Service Accounts

• If creating a new service account, click + CREATE SERVICE ACCOUNT. Provide a name for this account and a short description. Click CREATE AND CONTINUE

• Select the role BigQuery Admin from the dropdown. Conditions to lock down the service account further can optionally be added. Click CONTINUE and then DONE

• Using the newly created service account (or an existing service account with the BigQuery Admin permission granted), create a new key for the service account. Choose JSON format. This should download a file to your machine.

• Open the downloaded file and copy the contents into the Service account JSON directly.

Restrict permissions

You can optionally restrict the granted permissions to service account. The minimum permissions Orchestra requires to manage your BigQuery job is BigQuery Job User. However if you intend to perform additional actions as part of the query job, you may need to grant additional permissions.

Location

This is the default region in GCP where your stream(s) are located. For example, europe-west2. This is configurable at the task level when setting up a pipeline.

Jobs

Job Name	Description
Run query	Run a QueryJob in BigQuery