AWS ECS (Elastic Container Service)

Run arbitrary workloads on provisioned Cloud Compute Capacity in AWS

Type: Cloud Provider / Infrastructure

Website: https://aws.amazon.com/ecs/

General docs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html

API Docs: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/making-api-requests.html

Authentication

To connect ECS to Orchestra, you will need to follow the same process you normally do when dealing with service principals in AWS:

• An AWS IAM user with the required permissions for the job you wish to run

• An access key for the IAM user

• (Recommended) Orchestra requires the wildcard resource policy permission to DescribeTaskDefintions in your account. Therefore we recommend creating a separate AWS account for Orchestra to interact with.

Instructions

1. Navigate to the IAM console in the AWS account you wish to run the job in.

2. Attach the required permissions. See below for the required permissions for each job.

3. Generate a CLI access key for the User. More information is available in the AWS docs here. Once the access key is generated you can download a CSV containing your Access Key ID and Secret Access key.

Required Permissions

ECS run task. The following permissions are required. If you wish to add a resource restriction you can add one to "RunTask", "DescribeTasks", and "StopTask" actions. However the "DescribeTaskDefintions" action must have the wildcard resource "*".

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ecs:RunTask",
        "ecs:DescribeTasks",
        "ecs:StopTask",
        "ecs:DescribeTaskDefinition"
      ],
      "Resource": "*"
    }
  ]
}

Execution and Task Roles. If you have added execution and/or task roles to the ECS task definition then the below permissions are also required.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "iam:PassRole",
        "iam:GetRole",
      ],
      "Resource": [
        "<EXECUTE_ROLE_ARN>",
        "<TASK_ROLE_ARN>"
      ]
    }
  ]
}

dbt Core. If using ECS to run dbt Core, in order for Orchestra to gather dbt operation metadata, then the user will require S3 permissions (example below). More details about using ECS to trigger dbt Core can be found here.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "<BUCKET_ARN>/*",
        "<BUCKET_ARN>"
      ]
    }
  ]
}

Jobs

Job name	Description
ECS run task	Runs a standalone task in ECS