feat(storage): add shared DynamoDB runner storage - #5279
Draft
edersonbrilhante wants to merge 49 commits into
Draft
feat(storage): add shared DynamoDB runner storage#5279edersonbrilhante wants to merge 49 commits into
edersonbrilhante wants to merge 49 commits into
Conversation
…dules (#5257) ## Description - Keep `modules/runners` and stable `multi_runner_config` dispatch unchanged. Stable configurations retain their historical `module.runners["configuration"]` addresses and flat `runners_map` fields. - Add explicit opt-in through `experimental.multi_runner_config_v2`. Stable and experimental configurations can coexist when their keys do not overlap; duplicate keys are rejected. - Normalize stable v1 once for shared queues, webhook matching, and runner-binary discovery while routing only v2 configurations through `modules/runner-stack`. - Make `runner-stack` the provider-neutral control plane for scale-up, scale-down, pool, job retry, SSM housekeeping, common Lambda IAM, and runner-role ownership. - Keep EC2-specific launch templates, instance profiles, security groups, AMI/bootstrap resources, runner log groups, IAM fragments, and Lambda environment fragments under `modules/compute-providers/ec2`. - Define provider-owned runner-role requirements in EC2 and attach them to the common runner role in `runner-stack`, allowing future compute providers to supply different policies without duplicating the role lifecycle. - Replace flat runner-stack inputs with ownership-based nested objects. Logging configuration is grouped under `observability.logs`, including `level`, retention, encryption, class, and tags. - Pass the canonical `compute_provider.ec2` object and nested `runner`, `github`, `ssm`, and `observability` objects directly into the EC2 resource and runner-role policy modules instead of expanding them back into prefixed scalar inputs. - Layer module, shared-resource, component, subcomponent, and EC2 runtime tags with documented precedence; provider-required EC2 bootstrap tags retain final precedence. - Group experimental v2 outputs by ownership: `runner.role`, `scale_up.{lambda,log_group,role}`, `scale_down.{lambda,log_group,role}`, nullable `pool.{lambda,log_group,role}`, and provider-specific resources under `provider.<type>`. - Use caller-known optional wrappers for external AMI parameters and KMS keys. The wrapper determines Terraform graph shape while its `arn` leaf may remain unknown until apply. - Generate runner-stack, pool, job-retry, and EC2 IAM policies with `aws_iam_policy_document` and retain provider-policy merge behavior. - Document the experimental boundary, ownership model, plan-time wrapper pattern, phased migration, and nested output contract under the internal module documentation path. This draft is stacked on #5251 because the provider boundary consumes the experimental v2 normalization introduced there. Lambda/TypeScript terminology changes are tracked separately in #5258. ## Test Plan - `pre-commit run --all-files` — Terraform fmt, TFLint, validation, and merge-conflict checks passed. - `terraform test` in `modules/runner-stack` — 11 passed. - `terraform test` in `modules/multi-runner` — 7 passed. - `terraform test` in `modules/compute-providers/ec2` — 4 passed. - `terraform test` in `modules/compute-providers/ec2/runner-role` — 3 passed. - `terraform test` in `modules/runner-stack/pool` — 1 passed. - `terraform test` in `modules/runner-stack/job-retry` — 1 passed. - `terraform validate` in `modules/lambda` — passed. - Verified `modules/runners` has no diff from `origin/main`, stable v1 still dispatches only to `module.runners`, and only the experimental map dispatches to `module.runner_stacks`. - Verified computed external role, profile, AMI-parameter, managed-policy, and KMS ARN inputs plan successfully through the real wrapper fixture. No live AWS apply was performed. Terraform tests use mocked providers, and state migration is intentionally deferred to the later migration phase. ## Related Issues Closes #5252 Depends on #5251 --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Use orchestration_provider consistently with compute_provider, centralize nested-module validation with terraform_data preconditions, and split stable-v1 from experimental-v2 routing tests.
Contributor
Dependency ReviewThe following issues were found:
License Issueslambdas/libs/storage-providers/package.json
OpenSSF ScorecardScorecard details
Scanned Files
|
edersonbrilhante
force-pushed
the
refactor-webhook-matcher-config-storage
branch
from
August 19, 2026 14:13
0150fb2 to
365ec35
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an opt-in
aws_dynamodbrunner-config storage provider for experimental multi-runner v2 deployments.<prefix>-configfor durable global and entry-scoped configuration, including GitHub App credentials, webhook secret, matcher config, runner bootstrap config, and runner-group cache.<prefix>-runner-statefor TTL-backed one-time runner configuration plus provider-neutral runner lifecycle/inventory records.scopeandidkeys to isolate global, runner-entry, lifecycle, and compute-resource records without creating a table per runner entry.aws_ssm. The termination watcher intentionally remains on SSM in this PR.DynamoDB selection requires direct GitHub App values; externally managed
*_ssmreferences continue using the SSM provider so Terraform does not copy those external SecureStrings into state.Test Plan
tsc --noEmit: control-plane, storage-providers, and webhook passed.terraform validate: storage provider, multi-runner, and webhook modules passed.terraform fmt -check -recursiveandgit diff --checkpassed.Related Issues
None.