Sync All

Description

This job triggers a connector using this endpoint.

Use Cases

We recommend creating a Sync All Task for every "source" within Fivetran, which should itself have a single connector in Fivetran.

This way, you can use Orchestra to trigger your data ingestion on a manual or event based schedule. This has a number of advantages vs. using Fivetran's in-built scheduler:

• You can co-ordinate tasks outside of Fivetran, like other ingestion tools for connectors Fivetran doesn't support, using Orchestra

• When Fivetran syncs run, Data Warehouse cost is incurred. Running these operations on a schedule you set explicitly ensures these costs do not go out of hand

• We aggregate metadata from the Fivetran Task in the same place as the metadata from other operations in your Pipeline

Parameters

These parameters are required to run the Sync All Task

Name	Data type	Restrictions	Example
Connector ID	String	N.A.	roast_chicken

Error handling

Already running flow

In Fivetran a connector is either syncing or not. Connectors do not have individual runs. Therefore Orchestra will check the status of a connector before requesting a new sync of the connector. If the connector is already syncing Orchestra fails the task to alert you that it is already syncing. See here for more details about why this is, along with potential remedies.

API Requests

If we receive the following error codes from Fivetran, we'll raise an error and the task will move to a failed state.

Code	Description	Handling
401	Unauthorised	We will raise an error and parse the raw error message from the Fivetran response as the Orchestra message
404	Not Found	We will raise an error with the HTTP Reason as the Orchestra message
Other error code		We will raise an error with the HTTP Reason as the Orchestra message

Broken connector

Fivetran's connectors have the concept of a state. Fetching a connector's detail yields the following API response:

{
    "code": "Success",
    "data": {
        "id": "roast_chicken",
        "group_id": "large_elephant",
        "service": "google_sheets",
        "service_version": 1,
        "schema": "google_sheets.sheet_name",
        "connected_by": "stinky_durian",
        "created_at": "2023-03-27T19:03:14.779002Z",
        "succeeded_at": "2023-07-14T20:48:00.074Z",
        "failed_at": "2023-10-16T10:02:04.721Z",
        "paused": false,
        "pause_after_trial": false,
        "sync_frequency": 1440,
        "data_delay_threshold": 0,
        "data_delay_sensitivity": "NORMAL",
        "schedule_type": "auto",
        "status": {
            "setup_state": "broken",
            "schema_status": "ready",
            "sync_state": "syncing",
            "update_state": "on_schedule",
            "is_historical_sync": false,
            "tasks": [],
            "warnings": []
        },
        "config": {
            "sheet_id": "https://docs.google.com/spreadsheets/d/{sheet_id}/edit#gid=edit_ref",
            "named_range": "named_range_name",
            "authorization_method": "User OAuth",
            "last_synced_changes__utc_": "2023-08-10 08:55"
        }
    }
}

In this example, the "setup_state" is "broken". Under the scenario where the setup_state is anything other than "connected", Orchestra will not run the task.