diff --git a/README.md b/README.md index b17dfc3b86..f097d3ebd2 100644 --- a/README.md +++ b/README.md @@ -66,7 +66,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh) ## Requirements | Name | Version | -| ---- | ------- | +|------|---------| | [terraform](#requirement\_terraform) | >= 1.3.0 | | [aws](#requirement\_aws) | >= 6.33 | | [random](#requirement\_random) | ~> 3.0 | @@ -74,14 +74,14 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh) ## Providers | Name | Version | -| ---- | ------- | -| [aws](#provider\_aws) | 6.60.0 | -| [random](#provider\_random) | 3.9.0 | +|------|---------| +| [aws](#provider\_aws) | >= 6.33 | +| [random](#provider\_random) | ~> 3.0 | ## Modules | Name | Source | Version | -| ---- | ------ | ------- | +|------|--------|---------| | [ami\_housekeeper](#module\_ami\_housekeeper) | ./modules/ami-housekeeper | n/a | | [instance\_termination\_watcher](#module\_instance\_termination\_watcher) | ./modules/termination-watcher | n/a | | [runner\_binaries](#module\_runner\_binaries) | ./modules/runner-binaries-syncer | n/a | @@ -92,7 +92,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh) ## Resources | Name | Type | -| ---- | ---- | +|------|------| | [aws_sqs_queue.queued_builds](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | | [aws_sqs_queue.queued_builds_dlq](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | | [aws_sqs_queue_policy.build_queue_dlq_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy) | resource | @@ -103,7 +103,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh) ## Inputs | Name | Description | Type | Default | Required | -| ---- | ----------- | ---- | ------- | :------: | +|------|-------------|------|---------|:--------:| | [additional\_github\_apps](#input\_additional\_github\_apps) | Additional GitHub Apps for distributing API rate limit usage. Each must be installed on the same repos/orgs as the primary app. | 
list(object({
    key_base64          = optional(string)
    key_base64_ssm      = optional(object({ arn = string, name = string }))
    id                  = optional(string)
    id_ssm              = optional(object({ arn = string, name = string }))
    installation_id     = optional(string)
    installation_id_ssm = optional(object({ arn = string, name = string }))
  }))
| `[]` | no | | [ami](#input\_ami) | AMI configuration for the action runner instances. This object allows you to specify all AMI-related settings in one place.

Parameters:
- `filter`: Map of lists to filter AMIs by various criteria (e.g., { name = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-*"], state = ["available"] })
- `owners`: List of AMI owners to limit the search. Common values: ["amazon"], ["self"], or specific AWS account IDs
- `id_ssm_parameter_arn`: ARN of an SSM parameter containing the AMI ID. If specified, this overrides both AMI filter and parameter name
- `kms_key_arn`: Optional KMS key ARN if the AMI is encrypted with a customer managed key

Defaults to null, in which case the module falls back to individual AMI variables (deprecated). | 
object({
    filter               = optional(map(list(string)), { state = ["available"] })
    owners               = optional(list(string), ["amazon"])
    id_ssm_parameter_arn = optional(string, null)
    kms_key_arn          = optional(string, null)
  })
| `null` | no | | [ami\_housekeeper\_cleanup\_config](#input\_ami\_housekeeper\_cleanup\_config) | Configuration for AMI cleanup.

`amiFilters` - Filters to use when searching for AMIs to cleanup. Default filter for images owned by the account and that are available.
`dryRun` - If true, no AMIs will be deregistered. Default false.
`launchTemplateNames` - Launch template names to use when searching for AMIs to cleanup. Default no launch templates.
`maxItems` - The maximum number of AMIs that will be queried for cleanup. Default no maximum.
`minimumDaysOld` - Minimum number of days old an AMI must be to be considered for cleanup. Default 30.
`ssmParameterNames` - SSM parameter names to use when searching for AMIs to cleanup. This parameter should be set when using SSM to configure the AMI to use. Default no SSM parameters. | 
object({
    amiFilters = optional(list(object({
      Name   = string
      Values = list(string)
      })),
      [{
        Name : "state",
        Values : ["available"],
        },
        {
          Name : "image-type",
          Values : ["machine"],
      }]
    )
    dryRun              = optional(bool, false)
    launchTemplateNames = optional(list(string))
    maxItems            = optional(number)
    minimumDaysOld      = optional(number, 30)
    ssmParameterNames   = optional(list(string))
  })
| `{}` | no | @@ -250,7 +250,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh) ## Outputs | Name | Description | -| ---- | ----------- | +|------|-------------| | [binaries\_syncer](#output\_binaries\_syncer) | n/a | | [instance\_termination\_handler](#output\_instance\_termination\_handler) | n/a | | [instance\_termination\_watcher](#output\_instance\_termination\_watcher) | n/a | diff --git a/docs/rate-limits-and-tuning.md b/docs/rate-limits-and-tuning.md index 571fd96ff1..88c74bf6d5 100644 --- a/docs/rate-limits-and-tuning.md +++ b/docs/rate-limits-and-tuning.md @@ -52,7 +52,7 @@ Without a token cache, each runner also costs a `POST /app/installations/{id}/ac ### Distributing load across multiple GitHub Apps -Rate limits are per App installation and cannot be raised. To scale beyond one App's budget, configure extra Apps with `additional_github_apps`. The control-plane lambdas select one App per invocation, making the effective limit N × the per-App limit. +Rate limits are per App installation and cannot be raised. To scale beyond one App's budget, configure extra Apps with `additional_github_apps`. The control-plane lambdas select one App per invocation, making the effective limit N × the per-App limit. Selection prefers the App with the most rate-limit budget remaining, based on the `x-ratelimit-remaining` headers observed by the running Lambda container; Apps that hit a secondary rate limit are skipped for 60 seconds. > [!IMPORTANT] > Every additional App must be installed on the same organizations or repositories as the primary App. The module cannot verify this. A missing installation surfaces at runtime as installation lookup 404s on the fraction of invocations that select the misconfigured App, which is hard to trace back to the installation. diff --git a/lambdas/functions/control-plane/src/github/auth.test.ts b/lambdas/functions/control-plane/src/github/auth.test.ts index c2524503b7..04864be8f2 100644 --- a/lambdas/functions/control-plane/src/github/auth.test.ts +++ b/lambdas/functions/control-plane/src/github/auth.test.ts @@ -12,6 +12,8 @@ import { getStoredInstallationId, onRateLimit, onSecondaryRateLimit, + reportAppRateLimit, + reportAppSecondaryRateLimit, resetAppCredentialsCache, } from './auth'; import { describe, it, expect, beforeEach, vi } from 'vitest'; @@ -435,3 +437,82 @@ describe('Test getStoredInstallationId', () => { expect(result1).toBe(67890); }); }); + +describe('Test rate-limit aware app selection', () => { + const decryptedValue = 'decryptedValue'; + const b64 = Buffer.from(decryptedValue, 'binary').toString('base64'); + const app2IdParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_id`; + const app2KeyParam = `/actions-runner/${ENVIRONMENT}/additional_github_app_0_key_base64`; + + beforeEach(() => { + const mockedAuth = vi.fn(); + mockedAuth.mockResolvedValue({ token: 'token' }); + const mockWithHook = Object.assign(mockedAuth, { hook: vi.fn() }); + vi.mocked(createAppAuth).mockReturnValue(mockWithHook); + + process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME = `/actions-runner/${ENVIRONMENT}/additional_github_apps_manifest`; + mockedGetParameter.mockResolvedValue(JSON.stringify([{ idParamName: app2IdParam, keyParamName: app2KeyParam }])); + mockedGetParameters.mockResolvedValue( + new Map([ + [PARAMETER_GITHUB_APP_ID_NAME, GITHUB_APP_ID], + [PARAMETER_GITHUB_APP_KEY_BASE64_NAME, b64], + [app2IdParam, '2'], + [app2KeyParam, b64], + ]), + ); + + // Pin the random start offset to 0 so selection is deterministic. + vi.spyOn(Math, 'random').mockReturnValue(0); + }); + + it('selects the app with the most rate limit budget remaining', async () => { + reportAppRateLimit(0, 100); + reportAppRateLimit(1, 5000); + + const result = await createGithubAppAuth(undefined); + expect(result.appIndex).toBe(1); + }); + + it('assumes full budget for apps without observed state', async () => { + reportAppRateLimit(0, 100); + // App 1 has no observed state and is assumed full. + + const result = await createGithubAppAuth(undefined); + expect(result.appIndex).toBe(1); + }); + + it('skips an app cooling down after a secondary rate limit', async () => { + reportAppRateLimit(0, 100); + reportAppRateLimit(1, 5000); + reportAppSecondaryRateLimit(1); + + const result = await createGithubAppAuth(undefined); + expect(result.appIndex).toBe(0); + }); + + it('falls back to the most budget when every app is cooling down', async () => { + reportAppRateLimit(0, 100); + reportAppRateLimit(1, 5000); + reportAppSecondaryRateLimit(0); + reportAppSecondaryRateLimit(1); + + const result = await createGithubAppAuth(undefined); + expect(result.appIndex).toBe(1); + }); + + it('short-circuits to the primary app in single-app deployments', async () => { + delete process.env.PARAMETER_GITHUB_APPS_MANIFEST_NAME; + reportAppRateLimit(0, 0); + + const result = await createGithubAppAuth(undefined); + expect(result.appIndex).toBe(0); + }); + + it('respects an explicitly provided appIndex', async () => { + reportAppRateLimit(0, 5000); + reportAppRateLimit(1, 100); + + const result = await createGithubAppAuth(undefined, '', 1); + expect(result.appIndex).toBe(1); + }); +}); diff --git a/lambdas/functions/control-plane/src/github/auth.ts b/lambdas/functions/control-plane/src/github/auth.ts index b40b120bdd..6bf5220604 100644 --- a/lambdas/functions/control-plane/src/github/auth.ts +++ b/lambdas/functions/control-plane/src/github/auth.ts @@ -77,6 +77,66 @@ interface GitHubAppCredential { let appCredentialsPromise: Promise | null = null; +interface AppRateLimitState { + remaining: number; + cooldownUntil: number; +} + +// Last known primary rate limit remaining and secondary rate limit cooldown +// per app index. Fed by response headers and throttling callbacks; persists +// across invocations in a warm lambda so selection converges quickly. +const appRateLimitStates = new Map(); +const SECONDARY_RATE_LIMIT_COOLDOWN_MS = 60_000; + +export function reportAppRateLimit(appIndex: number, remaining: number): void { + const state = appRateLimitStates.get(appIndex) ?? { remaining, cooldownUntil: 0 }; + state.remaining = remaining; + appRateLimitStates.set(appIndex, state); +} + +export function reportAppSecondaryRateLimit(appIndex: number): void { + const state = appRateLimitStates.get(appIndex) ?? { remaining: 0, cooldownUntil: 0 }; + state.cooldownUntil = Date.now() + SECONDARY_RATE_LIMIT_COOLDOWN_MS; + appRateLimitStates.set(appIndex, state); + logger.warn(`GitHub App index ${appIndex} put in secondary rate limit cooldown`); +} + +// Select the app with the most primary rate limit budget remaining, skipping +// apps cooling down after a secondary rate limit. Apps with no observed state +// are assumed full. Iteration starts at a random offset so concurrent +// cold-started lambdas do not all converge on the same app. +async function selectAppIndex(): Promise { + const credentials = await getAppCredentials(); + if (credentials.length === 1) return 0; + const now = Date.now(); + const offset = Math.floor(Math.random() * credentials.length); + let best = -1; + let bestRemaining = -1; + for (let n = 0; n < credentials.length; n++) { + const i = (offset + n) % credentials.length; + const state = appRateLimitStates.get(i); + if (state && state.cooldownUntil > now) continue; + const remaining = state?.remaining ?? Number.MAX_SAFE_INTEGER; + if (remaining > bestRemaining) { + bestRemaining = remaining; + best = i; + } + } + if (best === -1) { + // Every app is cooling down; pick the one with the most remaining anyway. + for (let i = 0; i < credentials.length; i++) { + const remaining = appRateLimitStates.get(i)?.remaining ?? Number.MAX_SAFE_INTEGER; + if (remaining > bestRemaining) { + bestRemaining = remaining; + best = i; + } + } + } + // Info so the app selection distribution is observable at default log level. + logger.info(`Selected GitHub App index ${best} with ${bestRemaining} rate limit remaining`); + return best; +} + // One entry per additional app in the manifest parameter. The manifest keeps // the lambda environment size constant regardless of the number of apps: the // environment carries only the manifest's parameter name, and the manifest @@ -158,6 +218,7 @@ export async function getAppCount(): Promise { export function resetAppCredentialsCache(): void { appCredentialsPromise = null; + appRateLimitStates.clear(); } export async function getStoredInstallationId(appIndex: number): Promise { @@ -165,7 +226,7 @@ export async function getStoredInstallationId(appIndex: number): Promise { +export async function createOctokitClient(token: string, ghesApiUrl = '', appIndex?: number): Promise { const CustomOctokit = Octokit.plugin(retry, throttling); const ocktokitOptions: OctokitOptions = { auth: token, @@ -190,8 +251,29 @@ export async function createOctokitClient(token: string, ghesApiUrl = ''): Promi }, }, throttle: { - onRateLimit, - onSecondaryRateLimit, + onRateLimit: ( + retryAfter: number, + options: Required, + octokit: CoreOctokit, + retryCount: number, + ) => { + if (appIndex !== undefined) { + // Primary budget exhausted for this app; steer new flows elsewhere. + reportAppRateLimit(appIndex, 0); + } + return onRateLimit(retryAfter, options, octokit, retryCount); + }, + onSecondaryRateLimit: ( + retryAfter: number, + options: Required, + octokit: CoreOctokit, + retryCount: number, + ) => { + if (appIndex !== undefined) { + reportAppSecondaryRateLimit(appIndex); + } + return onSecondaryRateLimit(retryAfter, options, octokit, retryCount); + }, }, }); } @@ -201,8 +283,7 @@ export async function createGithubAppAuth( ghesApiUrl = '', appIndex?: number, ): Promise { - const credentials = await getAppCredentials(); - const idx = appIndex ?? Math.floor(Math.random() * credentials.length); + const idx = appIndex ?? (await selectAppIndex()); const auth = await createAuth(installationId, ghesApiUrl, idx); const result = await auth({ type: 'app' }); return { ...result, appIndex: idx }; @@ -213,8 +294,7 @@ export async function createGithubInstallationAuth( ghesApiUrl = '', appIndex?: number, ): Promise { - const credentials = await getAppCredentials(); - const idx = appIndex ?? Math.floor(Math.random() * credentials.length); + const idx = appIndex ?? (await selectAppIndex()); const auth = await createAuth(installationId, ghesApiUrl, idx); return auth({ type: 'installation', installationId }); } @@ -233,8 +313,7 @@ async function createAuth( appIndex?: number, ): Promise { const credentials = await getAppCredentials(); - const selected = - appIndex !== undefined ? credentials[appIndex] : credentials[Math.floor(Math.random() * credentials.length)]; + const selected = credentials[appIndex ?? (await selectAppIndex())]; logger.debug(`Selected GitHub App ${selected.appId} for authentication`); diff --git a/lambdas/functions/control-plane/src/github/octokit.ts b/lambdas/functions/control-plane/src/github/octokit.ts index 010516e436..46b292686c 100644 --- a/lambdas/functions/control-plane/src/github/octokit.ts +++ b/lambdas/functions/control-plane/src/github/octokit.ts @@ -68,7 +68,7 @@ export async function getInstallationId( appIndex?: number, ): Promise { const ghAuth = await createGithubAppAuth(undefined, ghesApiUrl, appIndex); - const githubClient = await createOctokitClient(ghAuth.token, ghesApiUrl); + const githubClient = await createOctokitClient(ghAuth.token, ghesApiUrl, appIndex); return resolveInstallationId(githubClient, enableOrgLevel, payload, appIndex); } @@ -88,13 +88,13 @@ export async function getOctokit( // Select one app for this entire auth flow const ghAuth = await createGithubAppAuth(undefined, ghesApiUrl); const appIdx = ghAuth.appIndex; - const githubAppClient = await createOctokitClient(ghAuth.token, ghesApiUrl); + const githubAppClient = await createOctokitClient(ghAuth.token, ghesApiUrl, appIdx); const installationId = await resolveInstallationId(githubAppClient, enableOrgLevel, payload, appIdx); try { const installationAuth = await createGithubInstallationAuth(installationId, ghesApiUrl, appIdx); - return await createOctokitClient(installationAuth.token, ghesApiUrl); + return await createOctokitClient(installationAuth.token, ghesApiUrl, appIdx); } catch (error) { // The installation id can be stale when it was reused from the webhook payload or from the // pre-configured per-app value while the app was uninstalled and reinstalled. Re-resolve the @@ -117,6 +117,6 @@ export async function getOctokit( }); const installationAuth = await createGithubInstallationAuth(resolvedInstallationId, ghesApiUrl, appIdx); - return await createOctokitClient(installationAuth.token, ghesApiUrl); + return await createOctokitClient(installationAuth.token, ghesApiUrl, appIdx); } } diff --git a/lambdas/functions/control-plane/src/github/rate-limit.test.ts b/lambdas/functions/control-plane/src/github/rate-limit.test.ts index 457a367c89..c3398d57e7 100644 --- a/lambdas/functions/control-plane/src/github/rate-limit.test.ts +++ b/lambdas/functions/control-plane/src/github/rate-limit.test.ts @@ -3,11 +3,12 @@ import { createSingleMetric } from '@aws-github-runner/aws-powertools-util'; import { MetricUnit } from '@aws-lambda-powertools/metrics'; import { metricGitHubAppRateLimit } from './rate-limit'; import { describe, it, expect, beforeEach, vi } from 'vitest'; -import { getLoadedAppId } from './auth'; +import { getLoadedAppId, reportAppRateLimit } from './auth'; vi.mock('./auth', async () => ({ // App ids per index, as loaded by the auth module from SSM. getLoadedAppId: vi.fn(async (appIndex: number) => [1234, 5678][appIndex]), + reportAppRateLimit: vi.fn(), })); vi.mock('@aws-github-runner/aws-powertools-util', async () => { @@ -91,6 +92,15 @@ describe('metricGitHubAppRateLimit', () => { }); }); + it('feeds the app selector with the remaining budget even when metrics are disabled', async () => { + process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'false'; + const headers: ResponseHeaders = { 'x-ratelimit-remaining': '4200', 'x-ratelimit-limit': '5000' }; + + await metricGitHubAppRateLimit(headers, 1); + + expect(reportAppRateLimit).toHaveBeenCalledWith(1, 4200); + }); + it('should label metric with an empty AppId when the appIndex is unknown', async () => { process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = 'true'; const headers: ResponseHeaders = { 'x-ratelimit-remaining': '75', 'x-ratelimit-limit': '5000' }; diff --git a/lambdas/functions/control-plane/src/github/rate-limit.ts b/lambdas/functions/control-plane/src/github/rate-limit.ts index 2cd194fbeb..2860905059 100644 --- a/lambdas/functions/control-plane/src/github/rate-limit.ts +++ b/lambdas/functions/control-plane/src/github/rate-limit.ts @@ -3,7 +3,7 @@ import { createSingleMetric, logger } from '@aws-github-runner/aws-powertools-ut import { MetricUnit } from '@aws-lambda-powertools/metrics'; import yn from 'yn'; -import { getLoadedAppId } from './auth'; +import { getLoadedAppId, reportAppRateLimit } from './auth'; // App ids come from the credentials already loaded by the auth module, so no // additional SSM reads are needed here. Index 0 is the primary app. @@ -19,6 +19,12 @@ export async function metricGitHubAppRateLimit(headers: ResponseHeaders, appInde logger.debug(`Rate limit remaining: ${remaining}, limit: ${limit}`); + // Feed the app selector so new auth flows prefer the app with the most + // budget left. Headers without an appIndex belong to the primary app. + if (!isNaN(remaining)) { + reportAppRateLimit(appIndex ?? 0, remaining); + } + const updateMetric = yn(process.env.ENABLE_METRIC_GITHUB_APP_RATE_LIMIT); if (updateMetric) { const appId = await getAppId(appIndex); diff --git a/lambdas/functions/control-plane/src/pool/pool.ts b/lambdas/functions/control-plane/src/pool/pool.ts index da5d2ee9b1..7d64aefc85 100644 --- a/lambdas/functions/control-plane/src/pool/pool.ts +++ b/lambdas/functions/control-plane/src/pool/pool.ts @@ -55,7 +55,7 @@ export async function adjust(event: PoolEvent): Promise { const installationId = await getInstallationId(ghAppAuth.token, ghesApiUrl, runnerOwner, appIdx); const ghAuth = await createGithubInstallationAuth(installationId, ghesApiUrl, appIdx); - const githubInstallationClient = await createOctokitClient(ghAuth.token, ghesApiUrl); + const githubInstallationClient = await createOctokitClient(ghAuth.token, ghesApiUrl, appIdx); // Get statuses of runners registered in GitHub const runnerStatusses = await getGitHubRegisteredRunnnerStatusses( @@ -120,7 +120,7 @@ async function getInstallationId(appToken: string, ghesApiUrl: string, org: stri const storedId = await getStoredInstallationId(appIndex); if (storedId !== undefined) return storedId; - const githubClient = await createOctokitClient(appToken, ghesApiUrl); + const githubClient = await createOctokitClient(appToken, ghesApiUrl, appIndex); return ( await githubClient.apps.getOrgInstallation({ diff --git a/lambdas/functions/control-plane/src/scale-runners/scale-down.ts b/lambdas/functions/control-plane/src/scale-runners/scale-down.ts index 3e387bce06..329ce694d9 100644 --- a/lambdas/functions/control-plane/src/scale-runners/scale-down.ts +++ b/lambdas/functions/control-plane/src/scale-runners/scale-down.ts @@ -41,7 +41,7 @@ async function getOrCreateOctokit(runner: RunnerInfo): Promise { // Use the pre-configured installation ID when available (avoids an API call). let installationId = await getStoredInstallationId(appIdx); if (installationId === undefined) { - const githubClientPre = await createOctokitClient(ghAuthPre.token, ghesApiUrl); + const githubClientPre = await createOctokitClient(ghAuthPre.token, ghesApiUrl, appIdx); installationId = runner.type === 'Org' ? ( @@ -57,7 +57,7 @@ async function getOrCreateOctokit(runner: RunnerInfo): Promise { ).data.id; } const ghAuth = await createGithubInstallationAuth(installationId, ghesApiUrl, appIdx); - const octokit = await createOctokitClient(ghAuth.token, ghesApiUrl); + const octokit = await createOctokitClient(ghAuth.token, ghesApiUrl, appIdx); githubCache.clients.set(key, octokit); return octokit; diff --git a/lambdas/functions/control-plane/src/scale-runners/scale-up.ts b/lambdas/functions/control-plane/src/scale-runners/scale-up.ts index 44d522a1f0..06ce69dc7c 100644 --- a/lambdas/functions/control-plane/src/scale-runners/scale-up.ts +++ b/lambdas/functions/control-plane/src/scale-runners/scale-up.ts @@ -44,7 +44,7 @@ async function createGithubInstallationClient( try { const ghAuth = await createGithubInstallationAuth(installationId, ghesApiUrl, appIndex); - return await createOctokitClient(ghAuth.token, ghesApiUrl); + return await createOctokitClient(ghAuth.token, ghesApiUrl, appIndex); } catch (error) { // The installation id can be stale when it was reused from the webhook payload or from the // pre-configured per-app value while the app was uninstalled and reinstalled. Re-resolve the @@ -67,7 +67,7 @@ async function createGithubInstallationClient( }); const ghAuth = await createGithubInstallationAuth(resolvedInstallationId, ghesApiUrl, appIndex); - return await createOctokitClient(ghAuth.token, ghesApiUrl); + return await createOctokitClient(ghAuth.token, ghesApiUrl, appIndex); } } @@ -103,7 +103,7 @@ export async function scaleUp(payloads: ActionRequestMessageSQS[]): Promise [terraform](#requirement\_terraform) | >= 1.3 | | [aws](#requirement\_aws) | >= 6.33 | | [random](#requirement\_random) | ~> 3.0 | @@ -106,14 +106,14 @@ module "multi-runner" { ## Providers | Name | Version | -| ---- | ------- | -| [aws](#provider\_aws) | 6.60.0 | -| [random](#provider\_random) | 3.9.0 | +|------|---------| +| [aws](#provider\_aws) | >= 6.33 | +| [random](#provider\_random) | ~> 3.0 | ## Modules | Name | Source | Version | -| ---- | ------ | ------- | +|------|--------|---------| | [ami\_housekeeper](#module\_ami\_housekeeper) | ../ami-housekeeper | n/a | | [instance\_termination\_watcher](#module\_instance\_termination\_watcher) | ../termination-watcher | n/a | | [runner\_binaries](#module\_runner\_binaries) | ../runner-binaries-syncer | n/a | @@ -124,7 +124,7 @@ module "multi-runner" { ## Resources | Name | Type | -| ---- | ---- | +|------|------| | [aws_sqs_queue.queued_builds](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | | [aws_sqs_queue.queued_builds_dlq](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource | | [aws_sqs_queue_policy.build_queue_dlq_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy) | resource | @@ -135,7 +135,7 @@ module "multi-runner" { ## Inputs | Name | Description | Type | Default | Required | -| ---- | ----------- | ---- | ------- | :------: | +|------|-------------|------|---------|:--------:| | [additional\_github\_apps](#input\_additional\_github\_apps) | Additional GitHub Apps for random API rate limit distribution.

The primary app (var.github\_app) is always included and is the one whose
webhook secret is used for incoming webhook signature validation. Only the
primary app needs a webhook configured in GitHub.

Additional apps listed here are used exclusively by the control-plane
lambdas (scale-up, scale-down, pool, job-retry) which randomly select an
app for each GitHub API call. Each additional app must be installed on the
same repositories/organizations as the primary app. | 
list(object({
    key_base64          = optional(string)
    key_base64_ssm      = optional(object({ arn = string, name = string }))
    id                  = optional(string)
    id_ssm              = optional(object({ arn = string, name = string }))
    installation_id     = optional(string)
    installation_id_ssm = optional(object({ arn = string, name = string }))
  }))
| `[]` | no | | [ami\_housekeeper\_cleanup\_config](#input\_ami\_housekeeper\_cleanup\_config) | Configuration for AMI cleanup. | 
object({
    maxItems       = optional(number)
    minimumDaysOld = optional(number)
    amiFilters = optional(list(object({
      Name   = string
      Values = list(string)
    })))
    launchTemplateNames = optional(list(string))
    ssmParameterNames   = optional(list(string))
    dryRun              = optional(bool)
  })
| `{}` | no | | [ami\_housekeeper\_lambda\_memory\_size](#input\_ami\_housekeeper\_lambda\_memory\_size) | Memory size limit in MB of the lambda. | `number` | `256` | no | @@ -219,7 +219,7 @@ module "multi-runner" { ## Outputs | Name | Description | -| ---- | ----------- | +|------|-------------| | [binaries\_syncer\_map](#output\_binaries\_syncer\_map) | n/a | | [instance\_termination\_handler](#output\_instance\_termination\_handler) | n/a | | [instance\_termination\_watcher](#output\_instance\_termination\_watcher) | n/a | diff --git a/modules/runners/README.md b/modules/runners/README.md index f5da55e112..05c6ea5620 100644 --- a/modules/runners/README.md +++ b/modules/runners/README.md @@ -51,27 +51,27 @@ yarn run dist ## Requirements | Name | Version | -| ---- | ------- | +|------|---------| | [terraform](#requirement\_terraform) | >= 1.3.0 | | [aws](#requirement\_aws) | >= 6.33 | ## Providers | Name | Version | -| ---- | ------- | -| [aws](#provider\_aws) | 6.60.0 | +|------|---------| +| [aws](#provider\_aws) | >= 6.33 | ## Modules | Name | Source | Version | -| ---- | ------ | ------- | +|------|--------|---------| | [job\_retry](#module\_job\_retry) | ./job-retry | n/a | | [pool](#module\_pool) | ./pool | n/a | ## Resources | Name | Type | -| ---- | ---- | +|------|------| | [aws_cloudwatch_event_rule.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule) | resource | | [aws_cloudwatch_event_rule.ssm_housekeeper](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_rule) | resource | | [aws_cloudwatch_event_target.scale_down](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_event_target) | resource | @@ -135,7 +135,7 @@ yarn run dist ## Inputs | Name | Description | Type | Default | Required | -| ---- | ----------- | ---- | ------- | :------: | +|------|-------------|------|---------|:--------:| | [ami](#input\_ami) | AMI configuration for the action runner instances. This object allows you to specify all AMI-related settings in one place.

Parameters:
- `filter`: Map of lists to filter AMIs by various criteria (e.g., { name = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-*"], state = ["available"] })
- `owners`: List of AMI owners to limit the search. Common values: ["amazon"], ["self"], or specific AWS account IDs
- `id_ssm_parameter_name`: Name of an SSM parameter containing the AMI ID. If specified, this overrides the AMI filter
- `id_ssm_parameter_arn`: ARN of an SSM parameter containing the AMI ID. If specified, this overrides both AMI filter and parameter name
- `kms_key_arn`: Optional KMS key ARN if the AMI is encrypted with a customer managed key

Defaults to null, in which case the module falls back to individual AMI variables (deprecated). | 
object({
    filter               = optional(map(list(string)), { state = ["available"] })
    owners               = optional(list(string), ["amazon"])
    id_ssm_parameter_arn = optional(string, null)
    kms_key_arn          = optional(string, null)
  })
| `null` | no | | [associate\_public\_ipv4\_address](#input\_associate\_public\_ipv4\_address) | Associate public IPv4 with the runner. Only tested with IPv4 | `bool` | `false` | no | | [aws\_partition](#input\_aws\_partition) | (optional) partition for the base arn if not 'aws' | `string` | `"aws"` | no | @@ -246,7 +246,7 @@ yarn run dist ## Outputs | Name | Description | -| ---- | ----------- | +|------|-------------| | [lambda\_pool](#output\_lambda\_pool) | n/a | | [lambda\_pool\_log\_group](#output\_lambda\_pool\_log\_group) | n/a | | [lambda\_scale\_down](#output\_lambda\_scale\_down) | n/a | diff --git a/modules/runners/job-retry/README.md b/modules/runners/job-retry/README.md index 267516fbbb..57c6d9dc91 100644 --- a/modules/runners/job-retry/README.md +++ b/modules/runners/job-retry/README.md @@ -11,26 +11,26 @@ The module is an inner module and used by the runner module when the opt-in feat ## Requirements | Name | Version | -| ---- | ------- | +|------|---------| | [terraform](#requirement\_terraform) | >= 1.3.0 | | [aws](#requirement\_aws) | >= 6.21 | ## Providers | Name | Version | -| ---- | ------- | +|------|---------| | [aws](#provider\_aws) | >= 6.21 | ## Modules | Name | Source | Version | -| ---- | ------ | ------- | +|------|--------|---------| | [job\_retry](#module\_job\_retry) | ../../lambda | n/a | ## Resources | Name | Type | -| ---- | ---- | +|------|------| | [aws_iam_role_policy.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy) | resource | | [aws_lambda_event_source_mapping.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_event_source_mapping) | resource | | [aws_lambda_permission.job_retry](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_permission) | resource | @@ -41,13 +41,13 @@ The module is an inner module and used by the runner module when the opt-in feat ## Inputs | Name | Description | Type | Default | Required | -| ---- | ----------- | ---- | ------- | :------: | +|------|-------------|------|---------|:--------:| | [config](#input\_config) | Configuration for the spot termination watcher lambda function.

`aws_partition`: Partition for the base arn if not 'aws'
`architecture`: AWS Lambda architecture. Lambda functions using Graviton processors ('arm64') tend to have better price/performance than 'x86\_64' functions.
`environment_variables`: Environment variables for the lambda.
`enable_organization_runners`: Enable organization runners.
`enable_metric`: Enable metric for the lambda. If `spot_warning` is set to true, the lambda will emit a metric when it detects a spot termination warning.
'ghes\_url': Optional GitHub Enterprise Server URL.
'user\_agent': Optional User-Agent header for GitHub API requests.
'github\_app\_parameters': Parameter Store for GitHub App Parameters.
'kms\_key\_arn': Optional CMK Key ARN instead of using the default AWS managed key.
`lambda_event_source_mapping_batch_size`: Maximum number of records to pass to the lambda function in a single batch for the event source mapping. When not set, the AWS default will be used.
`lambda_event_source_mapping_maximum_batching_window_in_seconds`: Maximum amount of time to gather records before invoking the lambda function, in seconds. AWS requires this to be greater than 0 if batch\_size is greater than 10.
`lambda_principals`: Add extra principals to the role created for execution of the lambda, e.g. for local testing.
`lambda_tags`: Map of tags that will be added to created resources. By default resources will be tagged with name and environment.
`log_level`: Logging level for lambda logging. Valid values are 'silly', 'trace', 'debug', 'info', 'warn', 'error', 'fatal'.
`logging_kms_key_id`: Specifies the kms key id to encrypt the logs with
`logging_retention_in_days`: Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653.
`memory_size`: Memory size limit in MB of the lambda.
`metrics`: Configuration to enable metrics creation by the lambda.
`prefix`: The prefix used for naming resources.
`role_path`: The path that will be added to the role, if not set the environment name will be used.
`role_permissions_boundary`: Permissions boundary that will be added to the created role for the lambda.
`runtime`: AWS Lambda runtime.
`s3_bucket`: S3 bucket from which to specify lambda functions. This is an alternative to providing local files directly.
`s3_key`: S3 key for syncer lambda function. Required if using S3 bucket to specify lambdas.
`s3_object_version`: S3 object version for syncer lambda function. Useful if S3 versioning is enabled on source bucket.
`security_group_ids`: List of security group IDs associated with the Lambda function.
'sqs\_build\_queue': SQS queue for build events to re-publish job request.
`subnet_ids`: List of subnets in which the action runners will be launched, the subnets needs to be subnets in the `vpc_id`.
`tag_filters`: Map of tags that will be used to filter the resources to be tracked. Only for which all tags are present and starting with the same value as the value in the map will be tracked.
`tags`: Map of tags that will be added to created resources. By default resources will be tagged with name and environment.
`timeout`: Time out of the lambda in seconds.
`tracing_config`: Configuration for lambda tracing.
`zip`: File location of the lambda zip file. | 
object({
    aws_partition               = optional(string, null)
    architecture                = optional(string, null)
    enable_organization_runners = bool
    environment_variables       = optional(map(string), {})
    ghes_url                    = optional(string, null)
    user_agent                  = optional(string, null)
    github_app_parameters = object({
      key_base64 = map(string)
      id         = map(string)
      additional_apps_manifest = optional(object({
        name = string
        arn  = string
      }), null)
      additional_app_parameter_arns = optional(list(string), [])
    })
    kms_key_arn                                                    = optional(string, null)
    lambda_event_source_mapping_batch_size                         = optional(number, 10)
    lambda_event_source_mapping_maximum_batching_window_in_seconds = optional(number, 0)
    lambda_tags                                                    = optional(map(string), {})
    log_level                                                      = optional(string, null)
    logging_kms_key_id                                             = optional(string, null)
    logging_retention_in_days                                      = optional(number, null)
    memory_size                                                    = optional(number, null)
    metrics = optional(object({
      enable    = optional(bool, false)
      namespace = optional(string, null)
      metric = optional(object({
        enable_github_app_rate_limit = optional(bool, true)
        enable_job_retry             = optional(bool, true)
      }), {})
    }), {})
    prefix = optional(string, null)
    principals = optional(list(object({
      type        = string
      identifiers = list(string)
    })), [])
    queue_encryption = optional(object({
      kms_data_key_reuse_period_seconds = optional(number, null)
      kms_master_key_id                 = optional(string, null)
      sqs_managed_sse_enabled           = optional(bool, true)
    }), {})
    role_path                 = optional(string, null)
    role_permissions_boundary = optional(string, null)
    runtime                   = optional(string, null)
    security_group_ids        = optional(list(string), [])
    subnet_ids                = optional(list(string), [])
    s3_bucket                 = optional(string, null)
    s3_key                    = optional(string, null)
    s3_object_version         = optional(string, null)
    sqs_build_queue = object({
      url = string
      arn = string
    })
    tags    = optional(map(string), {})
    timeout = optional(number, 30)
    tracing_config = optional(object({
      mode                  = optional(string, null)
      capture_http_requests = optional(bool, false)
      capture_error         = optional(bool, false)
    }), {})
    zip = optional(string, null)
  })
| n/a | yes | ## Outputs | Name | Description | -| ---- | ----------- | +|------|-------------| | [job\_retry\_check\_queue](#output\_job\_retry\_check\_queue) | n/a | | [lambda](#output\_lambda) | n/a | diff --git a/modules/runners/pool/README.md b/modules/runners/pool/README.md index 5a1ff2edcf..54b85d968e 100644 --- a/modules/runners/pool/README.md +++ b/modules/runners/pool/README.md @@ -9,14 +9,14 @@ The pool is an opt-in feature. To be able to use the count on a module level to ## Requirements | Name | Version | -| ---- | ------- | +|------|---------| | [terraform](#requirement\_terraform) | >= 0.14.1 | | [aws](#requirement\_aws) | >= 6.21 | ## Providers | Name | Version | -| ---- | ------- | +|------|---------| | [aws](#provider\_aws) | >= 6.21 | ## Modules @@ -26,7 +26,7 @@ No modules. ## Resources | Name | Type | -| ---- | ---- | +|------|------| | [aws_cloudwatch_log_group.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_group) | resource | | [aws_iam_role.pool](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | | [aws_iam_role.scheduler](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource | @@ -47,7 +47,7 @@ No modules. ## Inputs | Name | Description | Type | Default | Required | -| ---- | ----------- | ---- | ------- | :------: | +|------|-------------|------|---------|:--------:| | [aws\_partition](#input\_aws\_partition) | (optional) partition for the arn if not 'aws' | `string` | `"aws"` | no | | [config](#input\_config) | Lookup details in parent module. | 
object({
    lambda = object({
      log_level                      = string
      logging_retention_in_days      = number
      logging_kms_key_id             = string
      log_class                      = string
      reserved_concurrent_executions = number
      s3_bucket                      = string
      s3_key                         = string
      s3_object_version              = string
      security_group_ids             = list(string)
      runtime                        = string
      architecture                   = string
      memory_size                    = number
      timeout                        = number
      zip                            = string
      subnet_ids                     = list(string)
      parameter_store_tags           = string
    })
    tags = map(string)
    ghes = object({
      url        = string
      ssl_verify = string
    })
    github_app_parameters = object({
      key_base64 = map(string)
      id         = map(string)
      additional_apps_manifest = optional(object({
        name = string
        arn  = string
      }), null)
      additional_app_parameter_arns = optional(list(string), [])
    })
    subnet_ids = list(string)
    runner = object({
      disable_runner_autoupdate            = bool
      ephemeral                            = bool
      enable_jit_config                    = bool
      enable_on_demand_failover_for_errors = list(string)
      scale_errors                         = list(string)
      boot_time_in_minutes                 = number
      labels                               = list(string)
      launch_template = object({
        name = string
      })
      group_name  = string
      name_prefix = string
      pool_owner  = string
      role = object({
        arn = string
      })
      use_dedicated_host = bool
    })
    runners_maximum_count         = number
    instance_types                = list(string)
    instance_type_priorities      = optional(map(number))
    instance_target_capacity_type = string
    instance_allocation_strategy  = string
    instance_max_spot_price       = string
    prefix                        = string
    pool = list(object({
      schedule_expression          = string
      schedule_expression_timezone = string
      size                         = number
    }))
    include_busy_runners                 = bool
    role_permissions_boundary            = string
    kms_key_arn                          = string
    ami_kms_key_arn                      = string
    ami_id_ssm_parameter_arn             = string
    role_path                            = string
    ssm_token_path                       = string
    ssm_config_path                      = string
    ami_id_ssm_parameter_name            = string
    ami_id_ssm_parameter_read_policy_arn = string
    arn_ssm_parameters_path_config       = string
    lambda_tags                          = map(string)
    user_agent                           = string
  })
| n/a | yes | | [tracing\_config](#input\_tracing\_config) | Configuration for lambda tracing. | 
object({
    mode                  = optional(string, null)
    capture_http_requests = optional(bool, false)
    capture_error         = optional(bool, false)
  })
| `{}` | no | @@ -55,7 +55,7 @@ No modules. ## Outputs | Name | Description | -| ---- | ----------- | +|------|-------------| | [lambda](#output\_lambda) | n/a | | [lambda\_log\_group](#output\_lambda\_log\_group) | n/a | | [role\_pool](#output\_role\_pool) | n/a | diff --git a/modules/ssm/README.md b/modules/ssm/README.md index 73ba9a2d09..66bce354e2 100644 --- a/modules/ssm/README.md +++ b/modules/ssm/README.md @@ -8,15 +8,15 @@ This module is used for storing configuration of runners, registration tokens an ## Requirements | Name | Version | -| ---- | ------- | +|------|---------| | [terraform](#requirement\_terraform) | >= 1.3.0 | | [aws](#requirement\_aws) | >= 6.21 | ## Providers | Name | Version | -| ---- | ------- | -| [aws](#provider\_aws) | 6.60.0 | +|------|---------| +| [aws](#provider\_aws) | >= 6.21 | ## Modules @@ -25,7 +25,7 @@ No modules. ## Resources | Name | Type | -| ---- | ---- | +|------|------| | [aws_ssm_parameter.additional_github_app_id](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_parameter) | resource | | [aws_ssm_parameter.additional_github_app_installation_id](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_parameter) | resource | | [aws_ssm_parameter.additional_github_app_key_base64](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_parameter) | resource | @@ -37,7 +37,7 @@ No modules. ## Inputs | Name | Description | Type | Default | Required | -| ---- | ----------- | ---- | ------- | :------: | +|------|-------------|------|---------|:--------:| | [additional\_github\_apps](#input\_additional\_github\_apps) | Additional GitHub Apps for distributing API rate limit usage. | 
list(object({
    key_base64          = optional(string)
    key_base64_ssm      = optional(object({ arn = string, name = string }))
    id                  = optional(string)
    id_ssm              = optional(object({ arn = string, name = string }))
    installation_id     = optional(string)
    installation_id_ssm = optional(object({ arn = string, name = string }))
  }))
| `[]` | no | | [github\_app](#input\_github\_app) | GitHub app parameters, see your github app.
You can optionally create the SSM parameters yourself and provide the ARN and name here, through the `*_ssm` attributes.
If you chose to provide the configuration values directly here,
please ensure the key is the base64-encoded `.pem` file (the output of `base64 app.private-key.pem`, not the content of `private-key.pem`).
Note: the provided SSM parameters arn and name have a precedence over the actual value (i.e `key_base64_ssm` has a precedence over `key_base64` etc). | 
object({
    key_base64 = optional(string)
    key_base64_ssm = optional(object({
      arn  = string
      name = string
    }))
    id = optional(string)
    id_ssm = optional(object({
      arn  = string
      name = string
    }))
    webhook_secret = optional(string)
    webhook_secret_ssm = optional(object({
      arn  = string
      name = string
    }))
  })
| n/a | yes | | [kms\_key\_arn](#input\_kms\_key\_arn) | Optional CMK Key ARN to be used for Parameter Store. | `string` | `null` | no | @@ -47,7 +47,7 @@ No modules. ## Outputs | Name | Description | -| ---- | ----------- | +|------|-------------| | [additional\_app\_parameters](#output\_additional\_app\_parameters) | n/a | | [additional\_apps\_manifest](#output\_additional\_apps\_manifest) | n/a | | [parameters](#output\_parameters) | n/a |