Skip to content

AWS maintenance#1810

Draft
andrewkern wants to merge 3 commits intopopsim-consortium:mainfrom
andrewkern:aws_maint
Draft

AWS maintenance#1810
andrewkern wants to merge 3 commits intopopsim-consortium:mainfrom
andrewkern:aws_maint

Conversation

@andrewkern
Copy link
Member

this is a workflow that aims to securely deal with uploading new artifacts (genetic maps / annotations) to the AWS site.

  • Adds a local helper script (maintenance/upload_to_s3.sh) that validates uploads against the
    catalog before triggering the workflow
  • AWS credentials are now stored as GitHub repo secrets, accessible only through a protected
    environment requiring admin approval

A maintainer runs the local script pointing at a tarball and the catalog entry it corresponds to:

./maintenance/upload_to_s3.sh data/HapMapII_GRCh38.tar.gz HomSap genetic_map HapMapII_GRCh38

The script:

  1. Looks up the species, genetic map/annotation in the stdpopsim catalog
  2. Verifies the local file's SHA256 matches the catalog entry
  3. Creates a temporary draft GitHub release to transport the file
  4. Triggers the GitHub Actions workflow

The workflow then:

  1. Checks out the repo and re-validates against the catalog
  2. Downloads the asset and verifies SHA256 again
  3. Checks that the S3 destination doesn't already exist (no overwrites)
  4. Uploads to S3
  5. Cleans up the draft release

This is in a draft state currently and will need some review

@codecov
Copy link

codecov bot commented Mar 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.81%. Comparing base (8bc753e) to head (93d8169).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1810   +/-   ##
=======================================
  Coverage   99.81%   99.81%           
=======================================
  Files         142      142           
  Lines        4873     4873           
  Branches      472      472           
=======================================
  Hits         4864     4864           
  Misses          6        6           
  Partials        3        3           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@andrewkern andrewkern requested a review from nspope March 3, 2026 20:43
@andrewkern
Copy link
Member Author

tagging you @nspope -- if you could help to review this that would be amazing

@andrewkern
Copy link
Member Author

@petrelharp -- heads up, it looks like the pyslim docs site is returning a 404

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant