Integrations

How to connect to any data tool or infrastructure using Orchestra

In Orchestra, a connection to a third party data tool or piece of data infrastructure is called an "Integration".

These could be data ingestion tools like Fivetran or Stitch, data transformation tools like dbt Cloud, data warehouses like Snowflake or reverse ELT tools, to name but a few.

We also recognise the need to trigger and monitor core pieces of infrastructure within cloud providers such as AWS, GCP or Azure. We recognise that while stitching together cloud vendor-specific pieces of infrastructure is often quite straightforward, providing end-to-end visbility and orchestration across these systems and third party saas tools is a challenge.

Therefore, in Orchestra Integrations to cloud infrastructure components exist too, like GCP Datastream, AWS ECS/EKS/EC2 and Azure Virtual machines, for example.

Features

• Authentication: every Integration should have the concept of a user or Service Principle that enables a Data Governance first authentication and usage approach

• Secret encryption in transit and at rest

  • Optional: secrets checked for expiry / validity up to every hour to avoid pipelines breaking due to secret expiry / rotation

Integrations supported

Data ingestion

• Stitch

• Fivetran

• Airbyte Cloud

• Airbyte Server (beta)

• Matillion Data Productivity Cloud ("DPC") (beta)

Utility

• HTTP (beta)

• Render

• Python SDK (beta)

Business Intelligence

• Sigma (beta)

• Lightdash (beta)

• Hex (beta)

• Power BI (beta)

Workflow orchestration

• Airflow (beta)

• Prefect cloud

Streaming Platforms

• GCP - Datastream

Data transformation

• dbt Cloud

• Coalesce

Reverse ELT

• Census

• Hightouch

Data warehouse

• Snowflake

• Databricks (beta)

Cloud Provider

• AWS Glue (beta)

• AWS EC2 (Elastic Cloud Compute) (beta)

• Azure Data Factory

• Azure VMs (beta)