Datastream check stream status

Description

This job checks the status of the stream using this endpoint. If the stream is not in a "RUNNING" state, the Task will fail

Use Cases

We recommend creating a Datastream check stream status Task for every Datastream you use in GCP.

This way, you can ensure that at a bare minimum, the Datastream is in a healthy and unpaused state before running data transformations on data in Datastream destinations, or other potentially expensive operations. This also acts as a natural check on the Datastream in the same place as all your other data assets.

Additionally:

• You can co-ordinate tasks outside of Datastream, like batch ingestion tools such as Airbyte or Fivetran, or potentially other streams

• You get a log of helpful metadata stored in Orchestra for the Datastream regarding its status

• We aggregate metadata from GCP for Datastream in the same place as the metadata from other operations in your Pipeline

Parameters

These parameters are required to run the Datastream check stream status Task

Name	Data type	Restrictions	Example
Connection ID	GUID	N.A.	0368bfcb-946f-4d0f-8b14-2252cdc1f5e0
Workspace ID (Optional)	GUID	N.A.	0368bfcb-946f-4d0f-8b14-2252cdc1f5e0

Error handling

Stream Status (Stati?)

If we receive the following statuses from GCP, we'll raise an error and the task will move to a failed state.

State	Description	Handling
STATE_UNSPECIFIED	Unspecified stream state.	We will raise an error and store the state in the Orchestra message
NOT_STARTED	The stream has been created but has not yet started streaming data.	We will raise an error and store the state in the Orchestra message
RUNNING	The stream is running.	We will raise an error and store the state in the Orchestra message
PAUSED	The stream is paused.	We will raise an error and store the state in the Orchestra message
MAINTENANCE	The stream is in maintenance mode. Updates are rejected on the resource in this state.	We will raise an error and store the state in the Orchestra message
FAILED	The stream is experiencing an error that is preventing data from being streamed.	We will raise an error and store the state in the Orchestra message
FAILED_PERMANENTLY	The stream has experienced a terminal failure.	We will raise an error and store the state in the Orchestra message
STARTING	The stream is starting, but not yet running.	We will raise an error and store the state in the Orchestra message
DRAINING	The Stream is no longer reading new events, but still writing events in the buffer.	We will raise an error and store the state in the Orchestra message