Sync
Description
This job triggers a connector using this endpoint.
Use Cases
We recommend creating a Airbyte Sync Task for every "source" within Airbyte, which should itself have a single connector in Airbyte.
This way, you can use Orchestra to trigger your data ingestion on a manual or event based schedule. This has a number of advantages vs. using Stitch's in-built scheduler:
You can co-ordinate tasks outside of Airbyte, like other ingestion tools for connectors Airbyte doesn't support, using Orchestra
When Airbyte replication jobs run, Data Warehouse cost is incurred. Running these operations on a schedule you set explicitly ensures these costs do not go out of hand
We aggregate metadata from the AirbyteTask in the same place as the metadata from other operations in your Pipeline
Parameters
These parameters are required to run the Sync Task
| Name | Data type | Restrictions | Example |
|---|---|---|---|
Connection ID | GUID | N.A. | 0368bfcb-946f-4d0f-8b14-2252cdc1f5e0 |
Workspace ID (Optional) | GUID | N.A. | 0368bfcb-946f-4d0f-8b14-2252cdc1f5e0 |
Error handling
API Requests
If we receive the following error codes from Airbyte, we'll raise an error and the task will move to a failed state.
Code | Description | Handling |
401 | Unauthorised | We will raise an error and parse the raw error message from the Airbyte response as the Orchestra message |
404 | Not Found | We will raise an error with the HTTP Reason as the Orchestra message |
Other error code | We will raise an error with the HTTP Reason as the Orchestra message |
Job failure
When a connection is manually triggered this creates a new Job. Information about the job history can be found in the Airbyte docs here. Orchestra maps the sync status as follows:
| Orchestra Status | Airbyte Sync Status | Airbyte Description |
|---|---|---|
Succeeded | Succeeded | 100% of the data has been extracted and loaded to the destination |
Failed | Partially Succeeded | A subset of the data has been loaded to the destination |
Failed | Failed | None of the data has been loaded to the destination |
Cancelled | Cancelled | The sync was cancelled manually before finishing |
Running | Running | The sync is currently running |
Last updated