diff --git a/mlops-multi-account-cdk/mlops-sm-project-template/README.md b/mlops-multi-account-cdk/mlops-sm-project-template/README.md index 8925f9d7..350fdbfb 100644 --- a/mlops-multi-account-cdk/mlops-sm-project-template/README.md +++ b/mlops-multi-account-cdk/mlops-sm-project-template/README.md @@ -3,13 +3,6 @@ This repository contains the resources that are required to deploy the MLOps Fou - [MLOps Foundation SageMaker Project Template](#mlops-foundation-sagemaker-project-template) - [Solution Architecture](#solution-architecture) - - [Service Catalog Stack](#service-catalog-stack) - - [SageMaker Project Stack](#sagemaker-project-stack) - - [Shared Resources](#shared-resources) - - [Build App CI/CD Construct](#build-app-cicd-construct) - - [Deploy App CI/CD Construct](#deploy-app-cicd-construct) - - [CodeCommit Stack](#codecommit-stack) - - [Pipeline Stack](#pipeline-stack) - [Getting Started](#getting-started) - [Prerequisites](#prerequisites) - [Repository Structure](#repository-structure) @@ -25,120 +18,12 @@ This repository contains the resources that are required to deploy the MLOps Fou ![mlops project architecture](diagrams/MLOPs%20Foundation%20Architecture-mlops%20project%20cicd%20architecture.jpg) +For a detailed explaination of the solution architecture go to [Solution Architecture](docs/SOLUTION_ARCHITECTURE.md) -### Service Catalog Stack - -*This stack is only deployed in the DEV account* - -In this stack we define the Service Catalog Portfolio and Product that we want to use for SageMaker Project Template. - -The stack expects 4 cloudformation parameters: -- **Execution Role Arn:** This parameter is of type `AWS::SSM::Parameter::Value` so you should provide an `SSM Parameter Name` as an input; the default value is `"/mlops/role/lead"` which is deployed as part of **mlops infrastructure repository**. Ensure you have this parameter defined in your account before you deploy the solution. This is the role that would be given visibility to the Service Catalog Portfolio and Products. Products with tag `sagemaker:studio-visibility = true` will be visible in SageMaker Studio Domain. -- **Portfolio Name:** This parameter is of type `String` and has a default value of `SageMaker Organization Templates`. It will be used for Service Catalog Portfolio resource. -- **Portfolio Owner:** This parameter is of type `String` and has a default value of `administrator`. It will be used for Service Catalog Portfolio and Product resources. -- **Product Version:** This parameter is of type `String` and has a default value of `1.0`. It will be used for Service Catalog Product resource. - -The following resources are created: -- **Product Launch Role** is created which has permissions to create all the resources defined in SageMaker Project Stack and access to resources used in that stack i.e. SSM Parameters and access to CDK Assets S3 bucket (deployed as part of CDKToolkit stack). -- **A Service Catalog Product**, CloudFormation Product, for SageMaker Project Template as defined in [SageMaker Project Stack](#sagemaker-project-stack). The Product Launch Role is linked to this product. -- **A Service Catalog Portfolio**, the product would be associated to this portfolio and has visibility to sagemaker studio and linked to the Execution Role provided in the cloudformation parameters. -- **2 assets**, a zip for the **seed code** for each of the **build app** and the **deploy app**, stored in CDK Assets bucket, this bucket is created as part of **CDKToolkit stack**. A SSM Parameter is created for each asset's s3 key and also for the s3 bucket name. These parameters are used in [SageMaker Project Stack](#sagemaker-project-stack). The assets structure is documented in [MLOps Foundation Strategy](link-quip) -- **SSM Parameters** containig configuration about the target deployment accounts: **DEV**, **PREPROD** and **PROD** i.e. account id, region. These Parameters are used by the deploy app. - -Once this stack is deployed in the account, the template will be usable by Amazon SageMaker Studio Domain user with the role: **Execution Role** that was provided to this stack. - - -### SageMaker Project Stack -*This stack is stored as a service catalog product in the DEV Account and is visible in SageMaker Studio Domain* - -![project architecture](diagrams/MLOPs%20Foundation%20Architecture-sagemaker%20project%20architecture.jpg) - -The ML solutions' strategy defined in this stack uses two repositories setup: **(a)** building/training repository for training and batch inference ML pipeline development, and **(b)** deployment repository to promote the batch inference pipeline models or instantiate the real time endpoints. The second repository will incorporate also the testing methods including, integration test, stress test, or custom ML tests that the data scientists want to perform to ensure the robustness of the models. - -The CI/CD pipelines follow a single branch strategy with deployments to the accounts driven by commits to the `main` branch. The pipeline contains a stage for each account deployment. - -The stack expects 2 cloudformation parameters: -- **Project Name:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. It will be visible in SageMaker Studio Domain and be used to then tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectName`. -- **Project ID:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. When creating a project in SageMaker Studio Domain and it will be automatically generated and then used to tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectId`. - -Other parameters can be included and those will be visible in SageMaker Studio Domain during Project Creation. - -All resources deployed as part of this stack are tagged with those 2 parameters. - -This stack can be broken into 3 parts: -- [Shared resources with Cross Account Permissions](#shared-resources) -- [Build application resources and CI/CD pipline](#build-app-cicd-construct) -- [Deploy application resources and CI/CD pipeline](#deploy-app-cicd-construct) - -#### Shared Resources -The following resources are deployed as part of a SageMaker Project and are used by both build and deploy applications: -- **Artifact Bucket** and its **KMS key**, this bucket will be used to store SageMaker Pipeline's steps outputs and also the trained model for this project. -- **Model Package Group**, this group is created at this stack mainly to setup its policy and enable cross account access to the other deployment accounts. -- **Pipeline Bucket** and its **KMS key**, this bucket will be used by both CICD pipelines to store the pipelines artifacts. - -These resources are deployed to the DEV account with cross account access enabled for PREPROD and PROD; except for the pipeline bucket and its kms key, those are constraint to the DEV account. - -#### Build App CI/CD Construct -*The CICD pipeline in this construct only deploys to DEV* -![build app](diagrams/building.png) - -This construct contains resources that create a CI/CD pipeline to orchestrate a model training using SageMaker Pipelines starting from doing preprocessing jobs over the data and end by registering the trained model in a model package group in SageMaker Model Registry. After the pipeline finishes running successfuly, the model will have `Pending Manual Approval` status in SageMaker Model Registry. - -In `seed_code/build_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based and is expected to run inside a CodeBuild project. There is a `buildspec.yml` that describes the command that will be run. For more details about this base code, refer to `seed_code/build_app/README.md`. - -The following resources are created in this construct: -- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/build_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/build`. The stack expects these parameters to exist in the account. -- **CodeBuild Project**, this codebuild project is used to create/update/run the SageMaker Pipeline defined in the **build app repository** in `ml_pipelines/training/pipeline.py`. The project's **buildspec** uses the `buildspec.yml` defined in the repository as well. -- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. It has one additional stage, **Build** stage, which runs the **CodeBuild Project** defined above. - -**NOTE** The solution defined above only support a CICD pipeline that is linked to the **main** branch update events. If you want to have a **multi-branch approach** i.e. a CI/CD pipeline linked to the **develop** then you will need to duplicate the exisiting pipeline and ensure resources for each branch are **isolated** to not have impact on each other. You will also need to recosider the prefix strategy used in **Artifact Bucket**, created in [shared resources section](#shared-resources). - - - -#### Deploy App CI/CD Construct -*The CICD pipeline in this construct only deploy to DEV, PREPROD and PROD* -![build app](diagrams/deployment.png) - -This construct contains resources that create a CI/CD pipeline to automate the deployment of an Amazon SageMaker Endpoint for real time inference across multiple accounts. - -In `seed_code/deploy_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based **AWS CDK** application. For more details about this base code, refer to `seed_code/deploy_app/README.md`. - - -The following resources are created in this construct: -- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/deploy_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/deploy`. The stack expects these parameters to exist in the account. -- **EventBridge Rule**, this rule is linked to the model package group used for this SageMaker Project. It will be monitoring **Approved** and **Rejected** events and will trigger the deployment CI/CD pipeline on them. -- **CodeBuild Project for CDK Synth**, this codebuild project is used to synthesize the AWS CDK application defined in the **deploy app repository**. 3 Cloudformation Templates are expected to be generated, each template references a targeted deployment account/environment: DEV, PREPROD and PROD. -- **CodeBuild Project for CFN Nag**, this codebuild project is used to run `cfn-nag` over the cloudformation templates to ensure that they satisfy best Security practices. -- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. The stages for this pipeline are defined as such: - - A **Build** stage which runs the **CodeBuild Project for CDK Synth** - - A **Security Evaluation** stage which runs the **CodeBuild Project for CFN Nag** - - There is a stage for each account: **DEV**, **PREPROD** and **PROD** each using Cloudformation Action to deploy the templates generated in the **Build** stage. There is a manual approval action before **PREPROD** and **PROD** deployments. - -There are 2 way to trigger the deployment CI/CD Pipeline: -- **Model Events** - These are events which get triggered through a status change to the model package group in SageMaker Model Registry. -- **Code Events** - The pipeline is triggered on git update events over a specific branch, in this solution it is linked to the **main** branch. - -**Note:** For the deployment stages for **PREPROD** and **PROD**, the roles defined for cloudformation deployment in `mlops_sm_project_template/templates/constructs/deploy_pipeline_construct.py` lines 284-292 and lines 317-326 are created when the **PREPROD** and **PROD** are bootstrapped with CDK with trust policies for the deployment CI/CD pipeline account (**DEV** account in our solution); the roles must be created before deploying this stack to any account along with trust policies included between the accounts and the roles. If you can bootstrap those accounts for any reason you should ensure to create similar roles in each of those accounts and adding them to the lines mentioned above in the file. - -### CodeCommit Stack -*This stack is only needed if you want to handle deployments of this folder of the repository to be managed through a CICD pipeline.* - -This stack handles setting up an AWS CodeCommit repository for this folder of the repository. This repository will be used as the source for the CI/CD pipeline defined in [Pipeline Stack](#pipeline-stack). The repository will be named based on the value defined in `mlops_sm_project_template/config/constants.py` with this variable `CODE_COMMIT_REPO_NAME`. The repository will be intialised with a default branch as defined in the `constants.py` file under `PIPELINE_BRANCH` variable. - -### Pipeline Stack - -*This stack is only needed if you want to handle deployments of this folder of repository to be managed through a CICD pipeline. The pipeline is configured to deploy to 1 account: DEV and will deploy the service catalog stack to the target account* - -The CICD pipeline in this repository is setup to monitor an AWS CodeCommit repository as defined in [CodeCommit Stack](#codecommit-stack). - -If you are using other sources like github or bitbucket for your repository, you will need to modify the connection to the appropriate repository as defined in `mlops_sm_project_template/pipeline_stack.py`. This can be done using AWS CodeStar but must be setup on the account. - -Make sure the pipelines also point to your targeted branch; by default the pipeline is linked to `main` branch events, this is defined in the `constants.py` file under `PIPELINE_BRANCH` variable. - -`constants.py` also contains information about the target accounts you want to use for this repository CI/CD pipeline and the target deployment accounts: **DEV**, **PREPROD** and **PROD**, this information will also be deployed in SSM Parameter in the DEV account for the Deploy App CI/CD pipeline. - -The pipeline will deploy all stacks and resources to the appropriate accounts. - +The following templates are deployed as part of the solution: +- [Basic Project Template](docs/templates/BASIC_PROJECT_TEMPLATE.md) +- [Dynamic Account Project Template](docs/templates/DYNAMIC_ACCOUNT_PROJECT_TEMPLATE.md) +- [BYOC Project Template](docs/templates/BYOC_PROJECT_TEMPLATE.md) ## Getting Started diff --git a/mlops-multi-account-cdk/mlops-sm-project-template/docs/ADVANCED_TOPICS.md b/mlops-multi-account-cdk/mlops-sm-project-template/docs/ADVANCED_TOPICS.md new file mode 100644 index 00000000..fb94c5aa --- /dev/null +++ b/mlops-multi-account-cdk/mlops-sm-project-template/docs/ADVANCED_TOPICS.md @@ -0,0 +1,174 @@ +# Advanced topics +The topics defined here assume you have already deployed the solution once following the steps in the main [README](README.md) + +- [Advanced topics](#advanced-topics) + - [Create new template](#create-new-template) + - [Setup CodeCommit with this repository](#setup-codecommit-with-this-repository) + - [Test the created sagemaker templates](#test-the-created-sagemaker-templates) + - [Delete a Project](#delete-a-project) + - [Delete a Domain](#delete-a-domain) + +## Create new template +You can use `templates/basic_project_stack.py` as the bases to create your own template. You need to follow these steps to create your own template: +1. create a new file in `mlops_sm_project_template/templates` with the suffix `_stack.py` (this is very important as the code will look for files with that suffix when it comes to pushing the templates) +2. copy the following into the new created stack: +``` +from aws_cdk import ( + Stack, + Tags, +) + + +from constructs import Construct + +class MLOpsStack(Stack): + DESCRIPTION: str = "" + TEMPLATE_NAME: str = "" + + def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None: + super().__init__(scope, construct_id, **kwargs) + + # Define required parmeters + project_name = aws_cdk.CfnParameter( + self, + "SageMakerProjectName", + type="String", + description="The name of the SageMaker project.", + min_length=1, + max_length=32, + ).value_as_string + + project_id = aws_cdk.CfnParameter( + self, + "SageMakerProjectId", + type="String", + min_length=1, + max_length=16, + description="Service generated Id of the project.", + ).value_as_string + + Tags.of(self).add("sagemaker:project-id", project_id) + Tags.of(self).add("sagemaker:project-name", project_name) +``` +The 2 parameters defined here are mandatory for any template created for Sagemaker Projects. + +3. now add any resources that you would like to create as part of your template i.e. S3 bucket, a pipeline to deploy/build the model .. etc. + + +## Setup CodeCommit with this repository +You would wonder after you have cloned this repository and deployed the solution how would you then start to interact with your deployed CodeCommit repository and start using it as a main repository and push changes to it. You have 2 options for this: +1. Clone the created CodeCommit repository and start treating it seperately from this repository +2. Just use this folder as a repository + +For the second option, you can do the following (while you are in the folder `mlops-sm-project-template`): +``` +git init +``` +this will create a local git for this folder which would be separate from the main so you can treat it as any git repo and it would not impact the main repository git. So, add the CodeCommit Repository as a remote source: +``` +git remote add origin https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/mlops-sm-project-template +``` +Ensure you have configured your machine to connect to CodeCommit and make `git push` or `git pull` commands to it; follow [Step 3 from the AWS documentation](https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-https-unixes.html). + +Now you can interact with the CodeCommit repository as normal. You will need to do the following for the first commit: +``` +git add -A +git commit -m "first commit" +export AWS_PROFILE=mlops-governance +git push origin main +make init # this will enable precommit which will now block any further pushes to the main branch +``` + +Ensure that your git uses the branch name **main** by default, otherwise the push command might fail and you will need to create a main branch then push changes through it. + +If you want to push the changes you made back to the main repository this folder belongs to you can just run this command: +``` +rm -fr .git +``` +This will remove the git settings from this folder so it would go back to the main repository settings. You can then raise a PR to include your changes to the main repository in GitHub. + + +## Test the created sagemaker templates +***NOTE:** make sure to run `cdk synth` before running any of the commands defined below.* + +You will need to deploy the `service catalog stack` as that would setup your account with the required resources and ssm parameters before you can start testing your templates directly. If you don't have the service catalog stack already deployed in your account, you can achieve this by running the following command: +``` +cdk --app ./cdk.out/assembly-Personal deploy —all --profile mlops-dev +``` + +otherwise make sure you have these ssm parameters defined: +- in the dev account: + - /mlops/dev/account_id + - /mlops/code/seed_bucket + - /mlops/code/build + - /mlops/code/build/byoc + - /mlops/code/deploy +- in the preprod account: + - /mlops/preprod/account_id + - /mlops/preprod/region +- in the prod account: + - /mlops/prod/account_id + - /mlops/prod/region + +**OPTION 1** For quick testing of the sagemaker templates, you could deploy the json generated by CDK directly in your account by running the following command: +``` +aws cloudformation deploy \ + --template-file ./cdk.out/byoc-project-stack-dev.template.json \ + --stack-name byoc-project-stack-dev \ + --region eu-west-1 \ + --capabilities CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \ + --disable-rollback \ + --s3-bucket \ + --profile mlops-dev \ + --parameter-overrides \ + SageMakerProjectName=mlops-test-0 \ + SageMakerProjectId=sm12340 +``` +This command will deploy the byoc project stack if you want to deploy other templates just change the `--template-file`, if you want to create a new stack you can change the other fields as well. + +**OPTION 2** It is also possible to use CDK command for this exact purpose but this would require you to add the following to `app.py` file: +``` +from mlops_sm_project_template.templates.byoc_project_stack import MLOpsStack + +MLOpsStack( + app, + "test", + env=deployment_env, +) +``` +The run `cdk synth` and then run the following to deploy: +``` +cdk deploy test --parameters SageMakerProjectName=mlops-test \ + --parameters SageMakerProjectId=sm1234 --profile mlops-dev +``` + +## Delete a Project +Deleting a project is a very involved process especially if you want to perform a complete delete of all the resources that were deployed for this project and even more tricky if those resource were deployed in other accounts. + +## Delete a Domain +Deleting a domain through CloudFormation will always fail especially if you have users with running applications in it. To be able to delete a domain you would need to run the following using `boto3` (you will need to know the `domain_id`): +``` +domain_apps = sm_client.list_apps(DomainIdEquals=domain_id) + + for app in domain_apps["Apps"]: + response = sm_client.delete_app( + DomainId=app["DomainId"], + UserProfileName=app["UserProfileName"], + AppType=app["AppType"], + AppName=app["AppName"], + ) + logger.info(response) + + logger.info(f"finished deleting apps for {domain_id} SageMaker Domain") + + domain_users = sm_client.list_user_profiles(DomainIdEquals=domain_id) + + for user_profile in domain_users["UserProfiles"]: + response = sm_client.delete_user_profile( + DomainId=user_profile["DomainId"], + UserProfileName=user_profile["UserProfileName"], + ) + logger.info(response) + + logger.info(f"finished deleting user profiles for {domain_id} SageMaker Domain") +``` \ No newline at end of file diff --git a/mlops-multi-account-cdk/mlops-sm-project-template/docs/SOLUTION_ARCHITECTURE.md b/mlops-multi-account-cdk/mlops-sm-project-template/docs/SOLUTION_ARCHITECTURE.md new file mode 100644 index 00000000..35f42422 --- /dev/null +++ b/mlops-multi-account-cdk/mlops-sm-project-template/docs/SOLUTION_ARCHITECTURE.md @@ -0,0 +1,54 @@ +# Solution Architecture + +![mlops project architecture](../diagrams/MLOPs%20Foundation%20Architecture-mlops%20project%20cicd%20architecture.jpg) + +- [Solution Architecture](#solution-architecture) + - [Service Catalog Stack](#service-catalog-stack) + - [CodeCommit Stack](#codecommit-stack) + - [Pipeline Stack](#pipeline-stack) + +## Service Catalog Stack + +*This stack is only deployed in the DEV account* + +In this stack we define the Service Catalog Portfolio and Product that we want to use for SageMaker Project Template. + +The stack expects 4 cloudformation parameters: +- **Execution Role Arn:** This parameter is of type `AWS::SSM::Parameter::Value` so you should provide an `SSM Parameter Name` as an input; the default value is `"/mlops/role/lead"` which is deployed as part of **mlops infrastructure repository**. Ensure you have this parameter defined in your account before you deploy the solution. This is the role that would be given visibility to the Service Catalog Portfolio and Products. Products with tag `sagemaker:studio-visibility = true` will be visible in SageMaker Studio Domain. +- **Portfolio Name:** This parameter is of type `String` and has a default value of `SageMaker Organization Templates`. It will be used for Service Catalog Portfolio resource. +- **Portfolio Owner:** This parameter is of type `String` and has a default value of `administrator`. It will be used for Service Catalog Portfolio and Product resources. +- **Product Version:** This parameter is of type `String` and has a default value of `1.0`. It will be used for Service Catalog Product resource. + +The following resources are created: +- **Product Launch Role** is created which has permissions to create all the resources defined in SageMaker Project Stack and access to resources used in that stack i.e. SSM Parameters and access to CDK Assets S3 bucket (deployed as part of CDKToolkit stack). +- **A Service Catalog Product**, CloudFormation Product, for SageMaker Project Template as defined in [SageMaker Project Stack](#sagemaker-project-stack). The Product Launch Role is linked to this product. +- **A Service Catalog Portfolio**, the product would be associated to this portfolio and has visibility to sagemaker studio and linked to the Execution Role provided in the cloudformation parameters. +- **2 assets**, a zip for the **seed code** for each of the **build app** and the **deploy app**, stored in CDK Assets bucket, this bucket is created as part of **CDKToolkit stack**. A SSM Parameter is created for each asset's s3 key and also for the s3 bucket name. These parameters are used in [SageMaker Project Stack](#sagemaker-project-stack). The assets structure is documented in [MLOps Foundation blog](https://aws.amazon.com/blogs/machine-learning/mlops-foundation-roadmap-for-enterprises-with-amazon-sagemaker/) and can be found in `seed_code` folder for reference. +- **SSM Parameters** containig configuration about the target deployment accounts: **DEV**, **PREPROD** and **PROD** i.e. account id, region. These Parameters are used by the deploy app. + +Once this stack is deployed in the account, the template will be usable by Amazon SageMaker Studio Domain user with the role: **Execution Role** that was provided to this stack. + +This stack deploys the following templates: +- [Basic Project Template](templates/BASIC_PROJECT_TEMPLATE.md) +- [Dynamic Account Project Template](templates/DYNAMIC_ACCOUNT_PROJECT_TEMPLATE.md) +- [BYOC Project Template](templates/BYOC_PROJECT_TEMPLATE.md) + +## CodeCommit Stack +*This stack is only needed if you want to handle deployments of this folder of the repository to be managed through a CICD pipeline.* + +This stack handles setting up an AWS CodeCommit repository for this folder of the repository. This repository will be used as the source for the CI/CD pipeline defined in [Pipeline Stack](#pipeline-stack). The repository will be named based on the value defined in `mlops_sm_project_template/config/constants.py` with this variable `CODE_COMMIT_REPO_NAME`. The repository will be intialised with a default branch as defined in the `constants.py` file under `PIPELINE_BRANCH` variable. + + +## Pipeline Stack + +*This stack is only needed if you want to handle deployments of this folder of repository to be managed through a CICD pipeline. The pipeline is configured to deploy to 1 account: DEV and will deploy the service catalog stack to the target account* + +The CICD pipeline in this repository is setup to monitor an AWS CodeCommit repository as defined in [CodeCommit Stack](#codecommit-stack). + +If you are using other sources like github or bitbucket for your repository, you will need to modify the connection to the appropriate repository as defined in `mlops_sm_project_template/pipeline_stack.py`. This can be done using AWS CodeStar but must be setup on the account. + +Make sure the pipelines also point to your targeted branch; by default the pipeline is linked to `main` branch events, this is defined in the `constants.py` file under `PIPELINE_BRANCH` variable. + +`constants.py` also contains information about the target accounts you want to use for this repository CI/CD pipeline and the target deployment accounts: **DEV**, **PREPROD** and **PROD**, this information will also be deployed in SSM Parameter in the DEV account for the Deploy App CI/CD pipeline. + +The pipeline will deploy all stacks and resources to the appropriate accounts. diff --git a/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/BASIC_PROJECT_TEMPLATE.md b/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/BASIC_PROJECT_TEMPLATE.md new file mode 100644 index 00000000..176b83ad --- /dev/null +++ b/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/BASIC_PROJECT_TEMPLATE.md @@ -0,0 +1,75 @@ +# Basic Project Template + + +## Basic Project Stack +*This stack is stored as a service catalog product in the DEV Account and is visible in SageMaker Studio Domain* + +![project architecture](../../diagrams/MLOPs%20Foundation%20Architecture-sagemaker%20project%20architecture.jpg) + +The ML solutions' strategy defined in this stack uses two repositories setup: **(a)** building/training repository for training and batch inference ML pipeline development, and **(b)** deployment repository to promote the batch inference pipeline models or instantiate the real time endpoints. The second repository will incorporate also the testing methods including, integration test, stress test, or custom ML tests that the data scientists want to perform to ensure the robustness of the models. + +The CI/CD pipelines follow a single branch strategy with deployments to the accounts driven by commits to the `main` branch. The pipeline contains a stage for each account deployment. + +The stack expects 2 cloudformation parameters: +- **Project Name:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. It will be visible in SageMaker Studio Domain and be used to then tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectName`. +- **Project ID:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. When creating a project in SageMaker Studio Domain and it will be automatically generated and then used to tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectId`. + +Other parameters can be included and those will be visible in SageMaker Studio Domain during Project Creation. + +All resources deployed as part of this stack are tagged with those 2 parameters. + +This stack can be broken into 3 parts: +- **[Shared resources with Cross Account Permissions](#shared-resources)** +- **[Build application resources and CI/CD pipeline](#build-app-cicd-construct)** +- **[Deploy application resources and CI/CD pipeline](#deploy-app-cicd-construct)** + + +### Shared Resources +The following resources are deployed as part of a SageMaker Project and are used by both build and deploy applications: +- **Artifact Bucket** and its **KMS key**, this bucket will be used to store SageMaker Pipeline's steps outputs and also the trained model for this project. +- **Model Package Group**, this group is created at this stack mainly to setup its policy and enable cross account access to the other deployment accounts. +- **Pipeline Bucket** and its **KMS key**, this bucket will be used by both CICD pipelines to store the pipelines artifacts. + +These resources are deployed to the DEV account with cross account access enabled for PREPROD and PROD; except for the pipeline bucket and its kms key, those are constraint to the DEV account. + +### Build App CI/CD Construct +*The CICD pipeline in this construct only deploys to DEV* +![build app](../../diagrams/building.png) + +This construct contains resources that create a CI/CD pipeline to orchestrate a model training using SageMaker Pipelines starting from doing preprocessing jobs over the data and end by registering the trained model in a model package group in SageMaker Model Registry. After the pipeline finishes running successfuly, the model will have `Pending Manual Approval` status in SageMaker Model Registry. + +In `seed_code/build_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based and is expected to run inside a CodeBuild project. There is a `buildspec.yml` that describes the command that will be run. For more details about this base code, refer to `seed_code/build_app/README.md`. + +The following resources are created in this construct: +- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/build_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/build`. The stack expects these parameters to exist in the account. +- **CodeBuild Project**, this codebuild project is used to create/update/run the SageMaker Pipeline defined in the **build app repository** in `ml_pipelines/training/pipeline.py`. The project's **buildspec** uses the `buildspec.yml` defined in the repository as well. +- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. It has one additional stage, **Build** stage, which runs the **CodeBuild Project** defined above. + +**NOTE** The solution defined above only support a CICD pipeline that is linked to the **main** branch update events. If you want to have a **multi-branch approach** i.e. a CI/CD pipeline linked to the **develop** then you will need to duplicate the exisiting pipeline and ensure resources for each branch are **isolated** to not have impact on each other. You will also need to recosider the prefix strategy used in **Artifact Bucket**, created in [shared resources section](#shared-resources). + + + +### Deploy App CI/CD Construct +*The CICD pipeline in this construct only deploy to DEV, PREPROD and PROD* +![build app](../../diagrams/deployment.png) + +This construct contains resources that create a CI/CD pipeline to automate the deployment of an Amazon SageMaker Endpoint for real time inference across multiple accounts. + +In `seed_code/deploy_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based **AWS CDK** application. For more details about this base code, refer to `seed_code/deploy_app/README.md`. + + +The following resources are created in this construct: +- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/deploy_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/deploy`. The stack expects these parameters to exist in the account. +- **EventBridge Rule**, this rule is linked to the model package group used for this SageMaker Project. It will be monitoring **Approved** and **Rejected** events and will trigger the deployment CI/CD pipeline on them. +- **CodeBuild Project for CDK Synth**, this codebuild project is used to synthesize the AWS CDK application defined in the **deploy app repository**. 3 Cloudformation Templates are expected to be generated, each template references a targeted deployment account/environment: DEV, PREPROD and PROD. +- **CodeBuild Project for CFN Nag**, this codebuild project is used to run `cfn-nag` over the cloudformation templates to ensure that they satisfy best Security practices. +- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. The stages for this pipeline are defined as such: + - A **Build** stage which runs the **CodeBuild Project for CDK Synth** + - A **Security Evaluation** stage which runs the **CodeBuild Project for CFN Nag** + - There is a stage for each account: **DEV**, **PREPROD** and **PROD** each using Cloudformation Action to deploy the templates generated in the **Build** stage. There is a manual approval action before **PREPROD** and **PROD** deployments. + +There are 2 way to trigger the deployment CI/CD Pipeline: +- **Model Events** - These are events which get triggered through a status change to the model package group in SageMaker Model Registry. +- **Code Events** - The pipeline is triggered on git update events over a specific branch, in this solution it is linked to the **main** branch. + +**Note:** For the deployment stages for **PREPROD** and **PROD**, the roles defined for cloudformation deployment in `mlops_sm_project_template/templates/constructs/deploy_pipeline_construct.py` lines 284-292 and lines 317-326 are created when the **PREPROD** and **PROD** are bootstrapped with CDK with trust policies for the deployment CI/CD pipeline account (**DEV** account in our solution); the roles must be created before deploying this stack to any account along with trust policies included between the accounts and the roles. If you can bootstrap those accounts for any reason you should ensure to create similar roles in each of those accounts and adding them to the lines mentioned above in the file. diff --git a/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/BYOC_PROJECT_TEMPLATE.md b/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/BYOC_PROJECT_TEMPLATE.md new file mode 100644 index 00000000..4fff5571 --- /dev/null +++ b/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/BYOC_PROJECT_TEMPLATE.md @@ -0,0 +1,76 @@ +# Basic Project Template + +## Basic Project Stack +*This stack is stored as a service catalog product in the DEV Account and is visible in SageMaker Studio Domain* + +![project architecture](../../diagrams/MLOPs%20Foundation%20Architecture-sagemaker%20project%20architecture.jpg) + +The ML solutions' strategy defined in this stack uses two repositories setup: **(a)** building/training repository for training and batch inference ML pipeline development, and **(b)** deployment repository to promote the batch inference pipeline models or instantiate the real time endpoints. The second repository will incorporate also the testing methods including, integration test, stress test, or custom ML tests that the data scientists want to perform to ensure the robustness of the models. + +The CI/CD pipelines follow a single branch strategy with deployments to the accounts driven by commits to the `main` branch. The pipeline contains a stage for each account deployment. + +The stack expects 2 cloudformation parameters: +- **Project Name:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. It will be visible in SageMaker Studio Domain and be used to then tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectName`. +- **Project ID:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. When creating a project in SageMaker Studio Domain and it will be automatically generated and then used to tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectId`. + +Other parameters can be included and those will be visible in SageMaker Studio Domain during Project Creation. + +All resources deployed as part of this stack are tagged with those 2 parameters. + +This stack can be broken into 3 parts: + +- **[Shared resources with Cross Account Permissions](#shared-resources)** +- **[Build application resources and CI/CD pipeline](#build-app-cicd-construct)** +- **[Deploy application resources and CI/CD pipeline](#deploy-app-cicd-construct)** + + +### Shared Resources +The following resources are deployed as part of a SageMaker Project and are used by both build and deploy applications: +- **Artifact Bucket** and its **KMS key**, this bucket will be used to store SageMaker Pipeline's steps outputs and also the trained model for this project. +- **Model Package Group**, this group is created at this stack mainly to setup its policy and enable cross account access to the other deployment accounts. +- **Pipeline Bucket** and its **KMS key**, this bucket will be used by both CICD pipelines to store the pipelines artifacts. +- **ECR repository**, this repository will be used by both CICD pipelines to store the docker images and later on use them during the cross account deployment for inference. + +These resources are deployed to the DEV account with cross account access enabled for PREPROD and PROD; except for the pipeline bucket and its kms key, those are constraint to the DEV account. + +### Build App CI/CD Construct +*The CICD pipeline in this construct only deploys to DEV* +![build app](../../diagrams/building.png) + +This construct contains resources that create a CI/CD pipeline to orchestrate a model training using SageMaker Pipelines starting from doing preprocessing jobs over the data and end by registering the trained model in a model package group in SageMaker Model Registry. After the pipeline finishes running successfuly, the model will have `Pending Manual Approval` status in SageMaker Model Registry. + +In `seed_code/build_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based and is expected to run inside a CodeBuild project. There is a `buildspec.yml` that describes the command that will be run. For more details about this base code, refer to `seed_code/build_app/README.md`. + +The following resources are created in this construct: +- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/build_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/build`. The stack expects these parameters to exist in the account. +- **CodeBuild Project**, this codebuild project is used to create/update/run the SageMaker Pipeline defined in the **build app repository** in `ml_pipelines/training/pipeline.py`. The project's **buildspec** uses the `buildspec.yml` defined in the repository as well. +- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. It has one additional stage, **Build** stage, which runs the **CodeBuild Project** defined above. + +**NOTE** The solution defined above only support a CICD pipeline that is linked to the **main** branch update events. If you want to have a **multi-branch approach** i.e. a CI/CD pipeline linked to the **develop** then you will need to duplicate the exisiting pipeline and ensure resources for each branch are **isolated** to not have impact on each other. You will also need to recosider the prefix strategy used in **Artifact Bucket**, created in [shared resources section](#shared-resources). + + + +### Deploy App CI/CD Construct +*The CICD pipeline in this construct only deploy to DEV, PREPROD and PROD* +![build app](../../diagrams/deployment.png) + +This construct contains resources that create a CI/CD pipeline to automate the deployment of an Amazon SageMaker Endpoint for real time inference across multiple accounts. + +In `seed_code/deploy_app`, you will find the base code that would be setup when you create a new SageMaker Project. It is **python** based **AWS CDK** application. For more details about this base code, refer to `seed_code/deploy_app/README.md`. + + +The following resources are created in this construct: +- **CodeCommit Repository**, this repository is intialised with the code defined in `seed_code/deploy_app` which was the stored in an S3 bucket when the service catalog stack was deployed. The s3 bucket name and code zip key are stored in ssm parameters: `/mlops/code/seed_bucket` and `/mlops/code/deploy`. The stack expects these parameters to exist in the account. +- **EventBridge Rule**, this rule is linked to the model package group used for this SageMaker Project. It will be monitoring **Approved** and **Rejected** events and will trigger the deployment CI/CD pipeline on them. +- **CodeBuild Project for CDK Synth**, this codebuild project is used to synthesize the AWS CDK application defined in the **deploy app repository**. 3 Cloudformation Templates are expected to be generated, each template references a targeted deployment account/environment: DEV, PREPROD and PROD. +- **CodeBuild Project for CFN Nag**, this codebuild project is used to run `cfn-nag` over the cloudformation templates to ensure that they satisfy best Security practices. +- **CodePipeline Pipeline**, this pipeline has the **CodeCommit Repository** created in this construct as source and monitors the commits to the **main** branch. The stages for this pipeline are defined as such: + - A **Build** stage which runs the **CodeBuild Project for CDK Synth** + - A **Security Evaluation** stage which runs the **CodeBuild Project for CFN Nag** + - There is a stage for each account: **DEV**, **PREPROD** and **PROD** each using Cloudformation Action to deploy the templates generated in the **Build** stage. There is a manual approval action before **PREPROD** and **PROD** deployments. + +There are 2 way to trigger the deployment CI/CD Pipeline: +- **Model Events** - These are events which get triggered through a status change to the model package group in SageMaker Model Registry. +- **Code Events** - The pipeline is triggered on git update events over a specific branch, in this solution it is linked to the **main** branch. + +**Note:** For the deployment stages for **PREPROD** and **PROD**, the roles defined for cloudformation deployment in `mlops_sm_project_template/templates/constructs/deploy_pipeline_construct.py` lines 284-292 and lines 317-326 are created when the **PREPROD** and **PROD** are bootstrapped with CDK with trust policies for the deployment CI/CD pipeline account (**DEV** account in our solution); the roles must be created before deploying this stack to any account along with trust policies included between the accounts and the roles. If you can bootstrap those accounts for any reason you should ensure to create similar roles in each of those accounts and adding them to the lines mentioned above in the file. diff --git a/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/DYNAMIC_ACCOUNT_PROJECT_TEMPLATE.md b/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/DYNAMIC_ACCOUNT_PROJECT_TEMPLATE.md new file mode 100644 index 00000000..ef66bdef --- /dev/null +++ b/mlops-multi-account-cdk/mlops-sm-project-template/docs/templates/DYNAMIC_ACCOUNT_PROJECT_TEMPLATE.md @@ -0,0 +1,15 @@ +# Dynamic Account Project Template +This template mainly differs to the [Basic Project Template](BASIC_PROJECT_TEMPLATE.md) in the template parameters. The archietcture and other componenets are exactly the same. + +## Dynamic Project Stack + +![project architecture](../../diagrams/MLOPs%20Foundation%20Architecture-sagemaker%20project%20architecture.jpg) + +The stack expects 5 cloudformation parameters: +- **Project Name:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. It will be visible in SageMaker Studio Domain and be used to then tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectName`. +- **Project ID:** This parameter is of type `String` and is mandatory to be created for SageMaker Project. When creating a project in SageMaker Studio Domain and it will be automatically generated and then used to tag all the resources related to SageMaker and have them visible under the project tab. It must be named `SageMakerProjectId`. +- **Preprod Account ID** This parameter is of type `String` and is required when using this template. It will be used to setup the pipeline so it can deploy to a **preprod account** and configure cross account permissions to the right account. +- **Prod Account ID** This parameter is of type `String` and is required when using this template. It will be used to setup the pipeline so it can deploy to a **prod account** and configure cross account permissions to the right account. +- **Deployment Region** This parameter is of type `String` and is required when using this template. It will be used to setup the deployment region for both **preprod** and **prod** accounts and configure cross account permissions to the right region in the accounts. + +