release update (#91)

* add code repo options, and custom resource for ecr repos

* handle existing ecr repos on create
* allow retaining repos on update/replace and delete
* add codecommit and bitbucket options for build source
* allow user specified buildspec files

* fix image lifecycle policy puts

* use awslabs/amazon-ebs-autoscale

* use new amazon-ebs-autoscale repo
* update ebs-autoscale README
* remove ebs-autoscale code from this repo

* remove deprecated custom ami template

* Add s3:GetObject permission to EC2 instance role

Add s3:GetObject permission to EC2 instance role for retrieving objects from S3 buckets and open datasets.

* Add S3 open datasets parameter

Add a new S3 open data bucket ARNs parameter and instance profile permission for using shared datasets.

* use common container build template

* move container build template to templates/_common
* make batch job definition optional for container builds
* require specification of buildspec.yml for container builds
* create workflow-tool container buildspec file
* use container build template for nextflow
* create nextflow container buildspec file

* remove deprecated ami build

* use container name variable for nextflow image

* propagate TemplateRootUrl to nextflow resources

* fix job def output value

* scope down nextflow permissions

* scratch mount no longer needed

* fix sfn example url

resolve #90

Co-authored-by: paulu-aws <[email protected]>
Co-authored-by: Jay Wang <[email protected]>

Author: 3 people

docs/quick-start.md

@@ -60,7 +60,7 @@ Below are the stand-alone CloudFormation templates for each of the sub-stacks. T
 
 | Name | Description | Source | Launch Stack |
 | -- | -- | :--: | :--: |
-{{ cfn_stack_row("AWS Step Functions Example", "SfnExample", "step-functions/sfn-example.template.yaml", "Create a Step Functions State Machine, Batch Job Definitions, and container images to run an example genomics workflow") }}
+{{ cfn_stack_row("AWS Step Functions Example", "SfnExample", "step-functions/sfn-workflow.template.yaml", "Create a Step Functions State Machine, Batch Job Definitions, and container images to run an example genomics workflow") }}
 {{ cfn_stack_row("Cromwell Server", "CromwellServer", "cromwell/cromwell-server.template.yaml", "Create an EC2 instance and an IAM instance profile to run Cromwell") }}
 {{ cfn_stack_row("Nextflow Resources", "NextflowResources", "nextflow/nextflow-resources.template.yaml", "Create Nextflow specific resources needed to run on AWS: an S3 Bucket for nextflow config and workflows, AWS Batch Job Definition for a Nextflow head node, and an IAM role for the nextflow head node job") }}
 

src/containers/buildspec-nextflow.yml

@@ -0,0 +1,25 @@
+# CodeBuild buildspec file for creating container image for nextflow
+# assumes the following environment variables:
+# - PROJECT_BRANCH: git branch / tag / commit-id to build
+# - PROJECT_PATH: path in the source to navigate to prior to build
+# - REGISTRY: docker image registry (e.g. ECR) to push the container image to
+# - CONTAINER_NAME: name of the container
+# - AWS_REGION: (Provided by CodeBuild) region to use for ECR
+version: 0.2
+phases:
+  pre_build:
+    commands:
+      - git checkout $PROJECT_BRANCH
+      - cd $PROJECT_PATH
+  build:
+    commands:
+      - echo "Building container"
+      - docker build -t ${CONTAINER_NAME} .
+  post_build:
+    commands:
+      - echo "Tagging container image for ECR"
+      - docker tag ${CONTAINER_NAME} ${REGISTRY}/${CONTAINER_NAME}
+      - echo "Docker Login to ECR"
+      - $(aws ecr get-login --no-include-email --region ${AWS_REGION})
+      - echo "Pushing container images to ECR"
+      - docker push ${REGISTRY}/${CONTAINER_NAME}

src/containers/buildspec-workflow-tool.yml

@@ -0,0 +1,27 @@
+# CodeBuild buildspec file for creating container images for a workflow tool
+# assumes the following environment variables:
+# - PROJECT_BRANCH: git branch / tag / commit-id to build
+# - PROJECT_PATH: path in the source to navigate to prior to build
+# - REGISTRY: docker image registry (e.g. ECR) to push the container image to
+# - CONTAINER_NAME: name of the container
+# - AWS_REGION: (Provided by CodeBuild) region to use for ECR
+version: 0.2
+phases:
+  pre_build:
+    commands:
+      - git checkout $PROJECT_BRANCH
+      - cd $PROJECT_PATH
+      - cp -R ../_common .
+  build:
+    commands:
+      - echo "Building container"
+      - chmod +x _common/build.sh
+      - _common/build.sh ${CONTAINER_NAME}
+  post_build:
+    commands:
+      - echo "Tagging container image for ECR"
+      - docker tag ${CONTAINER_NAME} ${REGISTRY}/${CONTAINER_NAME}
+      - echo "Docker Login to ECR"
+      - $(aws ecr get-login --no-include-email --region ${AWS_REGION})
+      - echo "Pushing container images to ECR"
+      - docker push ${REGISTRY}/${CONTAINER_NAME}

src/ebs-autoscale/.gitignore

@@ -1,58 +1,6 @@
 # Amazon Elastic Block Store Autoscale
 
-This is an example of a small daemon process that monitors a BTRFS filesystem mountpoint and automatically expands it when free space falls below a configured threshold. New [Amazon EBS](https://aws.amazon.com/ebs/) volumes are added to the instance as necessary and the underlying [BTRFS filesystem](http://btrfs.wiki.kernel.org) expands while still mounted. As new devices are added, the BTRFS metadata blocks are rebalanced to mitigate the risk that space for metadata will not run out.
+## RELOCATION NOTICE
 
-## Assumptions:
-
-1. That this code is running on a AWS EC2 instance
-2. The instance has a IAM Instance Profile with appropriate permissions to create and attache new EBS volumes. Ssee the [IAM Instance Profile](#iam_instance_profile) section below for more details
-3. That prerequisites are installed on the instance.
-
-Provided in this repo are:
-
-1. A python [script](bin/create-ebs-volume.py) that creates and attaches new EBS volumes to the current instance
-2. The daemon [script](bin/ebs-autoscale) that monitors disk space and expands the BTRFS filesystem by leveraging the above script to add EBS volumes, expand the filesystem, and rebalance the metadata blocks
-2. A template for an [upstart configuration file](templates/ebs-autoscale.conf.template)
-2. A [logrotate configuration file](templates/ebs-autoscale.logrotate) which should not be needed but may as well be in place for long-running instances.
-5. A [initialization script](bin/init-ebs-autoscale.sh) to configure and install all of the above
-6. A [cloud-init](templates/cloud-init-userdata.yaml) file for user-data that installs required packages and runs the initialization script. By default this creates a mount point of `/scratch` on a encrypted 20GB EBS volume. To change the mount point, edit the file.
-
-## Installation
-
-The easiest way to set up an instance is to provide a launch call with the userdata [cloud-init script](templates/cloud-init-userdata.yaml). Here is an example of launching the [Amazon ECS-Optimized  AMI](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html) in us-east-1 using this file:
-
-```bash
-aws ec2 run-instances --image-id ami-5253c32d \
-  --key-name MyKeyPair \
-  --user-data file://./templates/cloud-init-userdata.yaml \
-  --count 1 \
-  --security-group-ids sg-123abc123 \
-  --instance-type t2.micro \
-  --iam-instance-profile Name=MyInstanceProfileWithProperPermissions
-```
-
-
-## A note on IAM Instance Profile
-
-In the above, we assume that the `MyInstanceProfileWithProperPermissions` EC2 Instance Profile exists and has the following permissions:
-
-```json
-{
-    "Version": "2012-10-17",
-    "Statement": [
-        {
-            "Effect": "Allow",
-            "Action": [
-                "ec2:AttachVolume",
-                "ec2:DescribeVolumeStatus",
-                "ec2:DescribeVolumes",
-                "ec2:ModifyInstanceAttribute",
-                "ec2:DescribeVolumeAttribute",
-                "ec2:CreateVolume",
-                "ec2:DeleteVolume"
-            ],
-            "Resource": "*"
-        }
-    ]
-}
-```
+The code for this daemon has been moved to the following repoository:
+[awslabs/amazon-ebs-autoscale](https://github.com/awslabs/amazon-ebs-autoscale)