EC2 SSM Run Script

Description

This job triggers a script to be run in your EC2 Instance

Use Cases

We recommend creating a EC2 SSM Run script for each of the scripts you wish to run in EC2

This way, you can use Orchestra to trigger running your script on a cron or event based schedule. This has a number of advantages:

• You can co-ordinate tasks outside of AWS

  • A common use case for this is to trigger a crawl job using that runs in EC2 and outputs data to S3. Once this job has completed Orchestra can trigger an ingestion job to move the data to Snowflake and then you can use a dbt job to transform that data correctly for you.

• You can use Orchestra to trigger jobs across AWS Accounts / Environments

• When AWS EC2 jobs run, cost is incurred. Running these operations on a schedule you set explicitly ensures these costs do not go out of hand

• We aggregate metadata from the AWS EC2 Task in the same place as the metadata from other operations in your Pipeline

Parameters and setup

These parameters are required to run the Run Workflow Task

Name	Data type	Restrictions	Example
InstanceID	String	N.A.	i-Kt94e289e818fa433
Command	String	separated with semicolon and new line for multiple commands	echo 'Hello World!'; sleep 10; echo 'Bye World!';

Configure SSM Agent on EC2

1. Setup an EC2 machine. You need the script you wish to run on the machine

2. Install SSM agent on the EC2 machine. You can either follow the quick setup guide here or you can manually install SSM Agent by following these instructions: Linux, MacOS or Windows.

3. You can test your command using Systems Manager

Example Command Blocks

echo 'Hello World!';
sleep 10;
echo 'Bye World!';

or

echo 'Running more advanced script';
sh /home/ubuntu/run.sh;

By using the multiline textarea you can add any number of commands you wish. The task will complete when all the commands have been run and have exited.

Output to S3

In order to save the standard output and error output from the script to S3 you must ensure you have the correct permissions configured on the IAM role used by Systems Manager to manage the instances. E.g.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObjectAcl", "s3:GetObject"],
      "Resource": "<bucket_arn>"
    }
  ]
}

Error handling

API Requests

TBU