🕶️Observability

How Orchestra surfaces metadata to enable rapid debugging

An Orchestration tool is the correct location to gather and aggregate metadata, which is known in industry as "Data Observability".

There are a number of advantages to having this type of information within an Orchestration tool:

1. A genuine focal point for monitoring and managing Pipelines; from Orchestra, users will be able to build / declare data pipelines, run them, and also see metadata relating to all tasks and data assets

2. A richer dataset; Orchestra correlates metadata to pipeline runs, so you can see detailed error messages and logs that relate to a specific Task Run or specific Task. Recent observability tools present this data in aggregate - showing the latest data. Correlating data to pipeline and task runs makes for an intuitive and rapid debugging experience

3. Efficiency; Orchestra collects the majority of metadata as part of running integration jobs anyway. The metadata is therefore collected in an extremely efficient way that doesn't require the drain on your API limits or cost to your data warehouse.

To understand what metadata is captured and how for what Tasks and Triggers, dive more into this section.