Task Groups
Run Tasks together in a collective fashion
Sometimes, you might want to group Tasks together and have subsequent tasks reference the Group, rather than every Task.
For example, suppose you want to run all of your data ingestion Tasks as a single Group, because the Transformation step depends on the successful ingestion of all the Tasks in the Group.
This would be quite arduous to do without Task Groups. This is because you need to manually specify the dependency to every task, which equates to drawing one line for every data ingestion task.
This can be illustrated with three data ingestion tasks and two data transformation tasks like so:
What would be simpler is to group the first three tasks and have subsequent tasks reference the group:
Expected Behaviour
The advantage of using Task Groups as user-facing features is that expected behaviour is very intuitive. There are no special properties of Task Groups (by default) as they are just a way for users to specify less dependencies. The behaviour of the two scenarios above is exactly the same - namely, if any task within the group fails (or any task a downstream task depends on fails), the downstream task will fail to executed and be skipped.
Orchestra UI
The Orchestra UI defaults to only using Task groups. This means you can easily add nodes to groups, but you can still just execute everything using Task Groups with only one Task in them to reproduce normal DAG behaviour.
Working from Right to Left
1 - Add a Task Group
By default, you can add a task group to any node by clicking the Add Task Button
2 - Add another Task Group
By selecting the three dots on any Task, you can add another separate Task Group to that node. If you click the three dots of any Task in a group of more than 1 Task (e.g. 3) then adding a new Task Group creates a dependency between the new Task Group and the Task Group of the selected node.
3 - A Task Group with Three Tasks
This Task Group has three Tasks that test different Snowflake tables' data quality. The Group is dependent on three Tasks, and will only run once those three Tasks are completed successfully. As the subsequent dbt Task depends on it, it is necessary for all Tasks in this group to complete successfully for the dbt Task to start (but not sufficient, since it also depends on a Fivetran Task that is separate).
4 - A Task to a Group
Every Task Group has a blue button called "Add Task". Here, there is a Task Group with only one Task in it. Clicking the "Add Task" button gives the user to add a task.
Last updated
