Skip to content

Conversation

@ConnorJC3
Copy link

Summary

Adds a new janitor feature to clean ECR container images from repos (does not delete the repos themselves). There is a limit of 10k images per repository, and every few months our CI hits this limit and we have to manually clean out the repo.

This janitor is disabled by default and must be enabled by CLI flag (1) to avoid breaking anyone's existing workflow and (2) because it is not possible to tag ECR repos, so it is not possible to skip repos with this cleaner, it will clean all ECR repos.

Also had to bump some AWS SDK dependencies to get this to work.

Testing

Dry run test:

$ go run ./cmd/aws-janitor --region us-west-2 --enable-ecr-images-clean=true --all=true --dry-run=true |& jq -r '.msg'
Regions: us-west-2
arn:aws:ecr:us-west-2:368597081700:repository/csi-node-driver-registrar: deleting 5 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-resizer: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/livenessprobe: deleting 5 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-attacher: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/snapshot-controller: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-provisioner: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-snapshotter: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/aws-ebs-csi-driver: deleting 2 images

Real delete test:

$ go run ./cmd/aws-janitor --region us-west-2 --enable-ecr-images-clean=true --all=true |& jq -r '.msg'               
Regions: us-west-2
arn:aws:ecr:us-west-2:368597081700:repository/csi-node-driver-registrar: deleting 5 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-resizer: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/livenessprobe: deleting 5 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-attacher: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/snapshot-controller: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-provisioner: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/csi-snapshotter: deleting 3 images
arn:aws:ecr:us-west-2:368597081700:repository/aws-ebs-csi-driver: deleting 2 images
$ aws ecr list-images --repository-name aws-ebs-csi-driver
{
    "imageIds": []
}

@k8s-ci-robot k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Aug 21, 2025
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ConnorJC3
Once this PR has been reviewed and has the lgtm label, please assign stevekuznetsov for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Aug 21, 2025
@k8s-ci-robot k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 21, 2025
@torredil
Copy link
Member

/lgtm

cc: @dims

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 26, 2025
@dims
Copy link
Member

dims commented Aug 26, 2025

@ConnorJC3 i understand and agree that false by default is helpful. but so it is not possible to skip repos with this cleaner, it will clean all ECR repos. can still lead to potentially bad outcomes, can you please add an allow list? only those specified in the allow list will be cleaned? (empty allow list means, don't do anything! so let's print a helpful message)

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 26, 2025
@ConnorJC3
Copy link
Author

@dims good idea, newest rev switches the flag from a binary true/false to a list of repos to clean

@torredil
Copy link
Member

torredil commented Sep 2, 2025

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 2, 2025
@ConnorJC3
Copy link
Author

/assign stevekuznetsov

Assigning @stevekuznetsov as k8s-ci-robot suggested, when you get a chance can you take a look? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants