GCP - BigQuery
Run queries in BigQuery
Last updated
Run queries in BigQuery
Last updated
Type: Warehouse
Website:
General docs:
API Docs:
To connect BigQuery to Orchestra, you will need:
a GCP account with BigQuery enabled in at least one project
the ability to create a service account in GCP, or create a new key for an existing service account
a dataset in BigQuery
Instructions
Orchestra connects to your GCP account via a service account. .
We recommend creating a new service account for Orchestra to use that only has the ability to access BigQuery in a given project. You can create a new service account in the GCP console. If you reuse an existing service account, you will need to ensure that it has the correct permissions to access BigQuery.
Navigate to
If creating a new service account, click + CREATE SERVICE ACCOUNT. Provide a name for this account and a short description. Click CREATE AND CONTINUE
Select the role BigQuery Admin from the dropdown. Conditions to lock down the service account further can optionally be added. Click CONTINUE and then DONE
Using the newly created service account (or an existing service account with the BigQuery Admin permission granted), create a new key for the service account. Choose JSON format. This should download a file to your machine.
Open the downloaded file and copy the contents into the Service account JSON directly.
You can optionally restrict the granted permissions to service account. The minimum permissions Orchestra requires to manage your BigQuery job is BigQuery Job User. However if you intend to perform additional actions as part of the query job, you may need to grant additional permissions.
This is the default region in GCP where your stream(s) are located. For example, europe-west2. This is configurable at the task level when setting up a pipeline.
Run query
Run a QueryJob in BigQuery