feat: Add flag for import existing resources

[electrofelix]

Pass through option for importing existing resources which allows using
Retain for DeletionPolicy on resources to avoid removal on data
resources if stack needs to be deleted and recreated.

Fixes: #1699

[electrofelix]

Main use case I have for this is that sometimes the stack gets into a funk and needs to be deleted and recreated, however if there are data resources in the same application stack specifying DeletionPolicy: Retain prevents re-deploying via sam.

The workarounds are to move data resources out to separate stacks, which can work for event buses and DBs, but is a pain when working with queues. End up needing to pass a multitude of resource identifiers in via parameters.

Using this approach, I can mark queues, event buses (+ archives), and any DBs to be retained and be able to teardown and redeploy without dropping messages/events/data, and at the same time avoid needing a complex orchastrated deployment spanning multiple repositories or needing complex CI jobs to handle what stacks to delete vs deploy.

Note: when testing I noticed CloudFormation doesn't handle automatically re-importing AWS::Logs::LogGroup resources particular well.

I have a fairly rough script that does the same as this code the AWS CLI, which encountered the same issue suggesting there may be a bug in Cloudformation. Appears it fails to remove some information from resources associated with the original stack meaning they still appear to be part of a stack and it refuses to import. Solving this in Cloudformation would automatically fix it for this feature as well.

It would be nicer for it to be available built in rather than needing to run the script sometimes:

TEMP_TEMPLATE=$(mktemp -t template.yaml) || exit 1

stack_name=$(toml get --toml-path ${configName} ${sectionName}.global.parameters.stack_name)
capabilities=$(toml get --toml-path ${configName} ${sectionName}.global.parameters.capabilities)
parameter_overrides=$(toml get --toml-path ${configName} ${sectionName}.global.parameters.parameter_overrides | sed -e 's/^\[//g; s/\]$//g')
params=""
for param in ${parameter_overrides}
do
    param=${param%,}
    param=${param%\'}
    param=${param#\'}
    key=${param%%=*}
    value=${param#*=}
    params="${params}${params:+ }ParameterKey=${key},ParameterValue=${value}"
done

# make sure all aws commands use the requested profile
export AWS_PROFILE=$profile

echo "Packaging stack and then using cloudformation to import existing using config file: '$configName' ..."
command="sam package --config-file $configName --output-template-file ${TEMP_TEMPLATE}"
echo "Executing $command"
$command


command="aws cloudformation describe-stacks --stack-name "${stack_name}""
echo "Checking if stack exists"
echo "Executing $command"
if ! $command >/dev/null 2>&1; then
    change_set_type=CREATE
    stack_complete_command=stack-create-complete
else
    change_set_type=UPDATE
    stack_complete_command=stack-update-complete
fi

change_set_name="change-set-$(date +%s)"
command="aws cloudformation create-change-set --stack-name ${stack_name} --change-set-type ${change_set_type} \
 --change-set-name ${change_set_name} --template-body "file://${TEMP_TEMPLATE}" --capabilities ${capabilities} \
 --parameters ${params} --import-existing-resources"
echo "Executing $command"
$command

sleep 2
echo "Waiting for changeset to be complete"
command="aws cloudformation wait change-set-create-complete --stack-name ${stack_name} --change-set-name ${change_set_name}"
echo "Executing $command"
$command
if [[ $? -ne 0 ]]; then
    failure_reason=$(aws cloudformation describe-change-set --stack-name ${stack_name} --change-set-name ${change_set_name} --query "StatusReason" --output text)
    if [[ "${failure_reason}" == *"The submitted information didn't contain changes."* ]] || [[ "${failure_reason}" == *"No updates are to be performed."* ]]; then
        echo "No changes detected, exiting"
        exit 0
    else
        echo "Changeset failed to create: ${failure_reason}"
        exit 1
    fi
fi
exit 0

[vicheey]

Thank you for your contribution, @electrofelix. The current code implementation is clear and concise. We just need to verify both forward and backward deployment scenarios that this update would not leave a stack in an unrecoverable state if deployment fail. Would you be able to enhance this further by adding integration tests to cover your specific deployment use case as well?

[electrofelix]

@vicheey sure, I presume it's just follow the existing integration tests such as https://github.com/aws/aws-sam-cli/blob/develop/tests/integration/deploy/test_deploy_command.py#L1542-L1593 going with the following steps:

• deploy template with some resources marked retain (eventbus + archive, sqs, & dynamodb)
• delete stack
• re-deploy same template and assert expected resources have import in deploy_process_execute.stdout.strip()

Presumably some work around tear will be needed for this scenario? Extend add_left_over_resources_from_stack to cover the additional resource types added?

Is this better as a separate test file to cover the import functionality under the deploy tests?

Might be a few weeks before I will get back to updating