Skip to content

feat: add optional DynamoDB runner config storage - #5266

Draft
wadherv wants to merge 1 commit into
github-aws-runners:mainfrom
wadherv:dynamodb_instead_of_ssm_parameter_store
Draft

feat: add optional DynamoDB runner config storage#5266
wadherv wants to merge 1 commit into
github-aws-runners:mainfrom
wadherv:dynamodb_instead_of_ssm_parameter_store

Conversation

@wadherv

@wadherv wadherv commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Related to: ssm-housekeeper throttled by DeleteParameter rate limit #5243
  • Add backward-compatible runner_config_storage support with ssm as the default backend and optional dynamodb storage for label-scoped runner config.
  • Create one DynamoDB runner config table per runner label/module with configurable billing mode, capacity, autoscaling, TTL, PITR, SSE, deletion protection, tags,
    and CloudWatch alarms.
  • Move label-scoped bootstrap config, runner group cache, and one-time token/JIT config to DynamoDB when enabled; common GitHub App/webhook config remains in SSM.
  • Update scale-up, pool, and runner startup scripts for Linux/macOS/Windows to use the selected backend.
  • Add DynamoDB Lambda client reuse with configurable retry mode, keep-alive, max sockets, and keep-alive timeout.
  • Add create flag for SSM housekeeping resources and align DynamoDB token TTL with SSM housekeeping retention.
  • Scope IAM permissions by backend and DynamoDB leading key prefixes.
  • Remove unnecessary DynamoDB item metadata fields.

Testing

  • node_modules/.bin/vitest run
  • node_modules/.bin/tsc -p functions/control-plane/tsconfig.json --noEmit
  • node_modules/.bin/eslint ...
  • node_modules/.bin/ncc build functions/control-plane/src/lambda.ts -o /private/tmp/control-plane-ncc-check
  • git diff --check

@wadherv
wadherv requested review from a team as code owners August 12, 2026 08:25
@wadherv
wadherv marked this pull request as draft August 12, 2026 08:26
@wadherv
wadherv force-pushed the dynamodb_instead_of_ssm_parameter_store branch from afa76c2 to eed4332 Compare August 12, 2026 08:27
@wadherv
wadherv force-pushed the dynamodb_instead_of_ssm_parameter_store branch from eed4332 to 661eb31 Compare August 12, 2026 08:49
@edersonbrilhante

Copy link
Copy Markdown
Contributor

@wadherv interesting idea. Since you added count in ssm, please add a file moved to move the resource state.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds optional per-label DynamoDB storage for runner bootstrap and registration configuration while retaining SSM as the default backend.

Changes:

  • Provisions configurable DynamoDB tables, alarms, autoscaling, encryption, and IAM policies.
  • Integrates backend selection into control-plane Lambdas and runner startup scripts.
  • Adds optional SSM housekeeper creation and multi-runner support.

Reviewed changes

Copilot reviewed 39 out of 42 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
variables.tf Exposes storage and housekeeper settings.
outputs.tf Exposes selected backend and table.
main.tf Passes storage and housekeeper configuration.
CHANGELOG.md Removes the 7.10.2 entry.
modules/runners/variables.tf Defines runner-module storage options.
modules/runners/main.tf Injects storage settings into userdata.
modules/runners/local.tf Derives backend keys and TTL values.
modules/runners/dynamodb.tf Provisions tables, items, scaling, and alarms.
modules/runners/runner-config.tf Makes SSM bootstrap parameters conditional.
modules/runners/logging.tf Stores CloudWatch configuration by backend.
modules/runners/scale-up.tf Configures DynamoDB for scale-up.
modules/runners/pool.tf Passes storage configuration to pool.
modules/runners/ssm-housekeeper.tf Adds conditional resource creation.
modules/runners/policies-runner.tf Selects runner IAM policy by backend.
modules/runners/outputs.tf Adds storage-related outputs.
modules/runners/templates/start-runner.sh Adds Linux DynamoDB retrieval.
modules/runners/templates/start-runner-osx.sh Adds macOS DynamoDB retrieval.
modules/runners/templates/start-runner.ps1 Adds Windows DynamoDB retrieval.
modules/runners/pool/variables.tf Extends pool configuration schema.
modules/runners/pool/main.tf Configures pool Lambda and IAM.
modules/runners/pool/policies/lambda-pool.json Scopes pool SSM access.
modules/runners/policies/lambda-scale-up.json Scopes scale-up SSM access.
modules/runners/policies/lambda-dynamodb-runner-config.json Adds Lambda DynamoDB permissions.
modules/runners/policies/instance-dynamodb-runner-config-policy.json Adds runner DynamoDB permissions.
modules/runners/policies/instance-cloudwatch-policy.json Makes SSM CloudWatch access conditional.
modules/multi-runner/variables.tf Adds per-runner storage configuration.
modules/multi-runner/runners.tf Passes storage and housekeeper settings.
modules/multi-runner/outputs.tf Exposes per-runner storage details.
lambdas/yarn.lock Locks new DynamoDB dependencies.
lambdas/libs/runner-providers/core/index.ts Adds shared storage types.
lambdas/functions/control-plane/package.json Adds DynamoDB SDK dependencies.
lambdas/functions/control-plane/src/modules.d.ts Declares storage environment variables.
lambdas/functions/control-plane/src/scale-runners/types.ts Re-exports storage types.
lambdas/functions/control-plane/src/scale-runners/scale-up.ts Loads storage configuration.
lambdas/functions/control-plane/src/scale-runners/runner-config-storage.ts Implements SSM and DynamoDB stores.
lambdas/functions/control-plane/src/scale-runners/runner-config-storage.test.ts Tests DynamoDB storage behavior.
lambdas/functions/control-plane/src/scale-runners/github-runner.ts Uses the selected storage backend.
lambdas/functions/control-plane/src/pool/pool.ts Loads storage settings for pools.
lambdas/functions/termination-watcher/src/types.d.ts Removes state-change event types.
lambdas/functions/termination-watcher/src/termination-warning.ts Narrows termination event handling.
lambdas/functions/termination-watcher/src/termination-warning.test.ts Removes state-change coverage.
lambdas/functions/termination-watcher/src/lambda.ts Narrows interruption handler typing.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

const isSpotInterruption = event['detail-type'] === 'EC2 Spot Instance Interruption Warning';
const metricName = isSpotInterruption && config.createSpotWarningMetric ? 'SpotInterruptionWarning' : undefined;
metricEvent(instance, event, metricName, logger);
metricEvent(instance, event, config.createSpotWarningMetric ? 'SpotInterruptionWarning' : undefined, logger);
Comment thread modules/runners/local.tf
Comment on lines +28 to +31
runner_config_dynamodb_token_leading_keys = (
var.runner_config_storage.dynamodb.token_key_prefix == null
? ["$${ec2:SourceInstanceARN}"]
: ["${var.runner_config_storage.dynamodb.token_key_prefix}*"]
Comment thread modules/runners/local.tf
Comment on lines +33 to +36
runner_config_dynamodb_ttl_seconds = coalesce(
var.runner_config_storage.dynamodb.token_ttl_seconds,
local.ssm_housekeeper.config.minimumDaysOld * 86400,
)
Comment on lines +724 to +727
validation {
condition = var.runner_config_storage.dynamodb.token_ttl_seconds == null ? true : var.runner_config_storage.dynamodb.token_ttl_seconds > 0
error_message = "`runner_config_storage.dynamodb.token_ttl_seconds` must be null or greater than zero."
}
Comment on lines +755 to +773
validation {
condition = var.runner_config_storage.dynamodb.client_max_attempts > 0
error_message = "`runner_config_storage.dynamodb.client_max_attempts` must be greater than zero."
}

validation {
condition = contains(["standard", "adaptive"], var.runner_config_storage.dynamodb.client_retry_mode)
error_message = "`runner_config_storage.dynamodb.client_retry_mode` must be either `standard` or `adaptive`."
}

validation {
condition = var.runner_config_storage.dynamodb.client_http_max_sockets > 0
error_message = "`runner_config_storage.dynamodb.client_http_max_sockets` must be greater than zero."
}

validation {
condition = var.runner_config_storage.dynamodb.client_http_keep_alive_msecs == null ? true : var.runner_config_storage.dynamodb.client_http_keep_alive_msecs >= 0
error_message = "`runner_config_storage.dynamodb.client_http_keep_alive_msecs` must be null or greater than or equal to zero."
}
user_errors = {
alarm_name = "${var.prefix}-runner-config-dynamodb-user-errors"
alarm_description = "DynamoDB user errors on the runner config table."
metric_name = "UserErrors"
@yalafe

yalafe commented Aug 19, 2026

Copy link
Copy Markdown

+1 - had the same pain. I independently built the same approach (per-instance token/config in DynamoDB with a TTL, read-then-delete mirroring the SSM lifecycle) and it held up in a live run with no SSM token writes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants