✏️Integration Jobs

The smallest unit of work

An integration job refers to a specific operation that can be performed within an integration.

For example, this might be triggering and monitoring for a sync in Stitch, or triggering and monitoring the movement of data in Fivetran.

Integration Jobs are predefined operations that require the bare minimum configuration to get started. This removes complexity for data teams, and typically data teams will only need to provide an ID or unique identifier to configure a job.

Example

Fivetran Sync All

In Fivetran, there is the concept of a connector which is a 1:1 mapping between a source (such as Salesforce) and a destination (such as Snowflake).

A connector can "sync". This entails moving the data from the source to the destination. A connector may contain the configuration for multiple data tables.

A "Fivetran Sync All" job triggers a Fivetran sync via this endpoint and monitors the result. It requires the Connector ID which is typically an arbitrarily chosen adjective and noun that have gained some internal notoriety at Fivetran (the holy grail of IDs supposedly "roast_chicken").

Supported Jobs

To see what jobs we support, head over to the relevant Integrations Page, and the Supported Integration Jobs will be listed as sub-directories.

Syncing Jobs

Some integration jobs are considered 'sync' jobs. This means that for the given object in the underlying platform, there is no 'Run' object. Ie the object is either syncing or not. This can cause problems in the Orchestra engine if a sync job takes longer than the configured schedule frequency, as the engine would request the platform re-trigger the object run before the first one has finished. This will create lots of Task Run objects in Orchestra that are forever stuck in a running state, and the next step in your Pipeline Run will never occur.

To avoid this problem Orchestra will check the state of the object in the underlying platform before triggering it. If Orchestra detects the object is in a Running state already, Orchestra will fail the task and alert you that the underlying object is already running so could not be started.

If you encounter this error Orchestra recommends you either change the amount of work being done by that job, or you decrease your Pipeline schedule's frequency.