BM-2466: Build Docker images once per production deployment#1654
BM-2466: Build Docker images once per production deployment#1654
Conversation
capossele
requested review from
a team,
Wollac,
austinabell,
ec2,
jonastheis,
nahoc,
unizarr,
willemolding and
zeroecco
as code owners
github-actions
bot
changed the title
willpote
left a comment
willpote
left a comment
There was a problem hiding this comment.
I think this is a good usecase for using stack references:
https://www.pulumi.com/tutorials/building-with-pulumi/stack-references/
Would simplify a lot of the pipeline shenanigans as we don't need to touch the build spec. Then we just try and get image output in the Pulumi typescript and use it if it exists
| const outputs: Record<string, pulumi.Output<string>> = {}; | ||
|
|
||
| // Primary image (every service has one) | ||
| const dockerfile = config.require('DOCKERFILE'); | ||
| const primaryImage = buildImage('primary', dockerfile, dockerTag); | ||
| outputs.imageRef = primaryImage.ref; | ||
|
|
||
| // Optional additional images (indexer has backfill + rewards) | ||
| const backfillDockerfile = config.get('DOCKERFILE_BACKFILL'); | ||
| if (backfillDockerfile) { | ||
| const backfillImage = buildImage('backfill', backfillDockerfile, `backfill-${dockerTag}`); | ||
| outputs.backfillImageRef = backfillImage.ref; | ||
| } | ||
|
|
||
| const rewardsDockerfile = config.get('DOCKERFILE_REWARDS'); | ||
| if (rewardsDockerfile) { | ||
| const rewardsImage = buildImage('rewards', rewardsDockerfile, `rewards-${dockerTag}`); | ||
| outputs.rewardsImageRef = rewardsImage.ref; | ||
| } |
There was a problem hiding this comment.
Its kind of awkward how we have this special case. Its possible to do lists in pulumi config (https://www.pulumi.com/docs/iac/concepts/config/#structured-configuration), so I'm imagining something like:
images:
paths:
- dockerfiles/indexer.dockerfile
- dockerfiles/rewards-indexer.dockerfile
- dockerfiles/market-indexer.dockerfile
Then we can parse the name of the image from the path, and set outputs[name] = image.ref
(or we could just use a comma separated list)
| if [ -n "$BUILDER_STACK" ]; then | ||
| echo "Dependent prod stack: reading image refs from builder stack $BUILDER_STACK" | ||
| IMAGE_REF=$(pulumi stack output imageRef --stack "$BUILDER_STACK" 2>/dev/null) || IMAGE_REF="" | ||
| [ -n "$IMAGE_REF" ] && pulumi config set IMAGE_URI "$IMAGE_REF" && echo "Set IMAGE_URI=$IMAGE_REF" || true | ||
| MARKET_REF=$(pulumi stack output marketImageRef --stack "$BUILDER_STACK" 2>/dev/null) || MARKET_REF="" | ||
| [ -n "$MARKET_REF" ] && pulumi config set MARKET_IMAGE_URI "$MARKET_REF" && echo "Set MARKET_IMAGE_URI" || true | ||
| REWARDS_REF=$(pulumi stack output rewardsImageRef --stack "$BUILDER_STACK" 2>/dev/null) || REWARDS_REF="" | ||
| [ -n "$REWARDS_REF" ] && pulumi config set REWARDS_IMAGE_URI "$REWARDS_REF" && echo "Set REWARDS_IMAGE_URI" || true | ||
| BACKFILL_REF=$(pulumi stack output backfillImageRef --stack "$BUILDER_STACK" 2>/dev/null) || BACKFILL_REF="" | ||
| [ -n "$BACKFILL_REF" ] && pulumi config set BACKFILL_IMAGE_URI "$BACKFILL_REF" && echo "Set BACKFILL_IMAGE_URI" || true | ||
| fi |
There was a problem hiding this comment.
If we use stack refs I don't think we need any of this
| // Dependent prod stacks receive BUILDER_STACK so the buildspec can read | ||
| // the pre-built image ref instead of rebuilding the Docker image. | ||
| if (builderStack) { | ||
| envVars.push({ | ||
| name: "BUILDER_STACK", | ||
| type: "PLAINTEXT", | ||
| value: builderStack, | ||
| }); | ||
| } |
There was a problem hiding this comment.
Also not needed with stack refs, reading stack is all done at runtime in pulumi
|
One more thing, maybe lets test this on Slasher pipeline before updating doing all of them? This sort of Pulumi stuff is hard to get right the first time, and will avoid us breaking every pipeline :D |
2 participants
Production pipelines currently rebuild the same Docker image independently for each chain stack (Base Sepolia, Eth Sepolia, Base Mainnet, etc.), wasting ~20-40 minutes of redundant Rust/Docker builds per service per deploy.
This PR designates
l-prod-84532(Base Sepolia) as the "builder" production stack. It builds the Docker image once and exports the image reference as a Pulumi stack output. Dependent production stacks (l-prod-8453,l-prod-11155111,l-prod-1) read this reference via pulumi stack output, set it as a config value, and skip their Docker build entirely.Changes:
Staging is unchanged, each staging stack still builds its own image. Manual pulumi up without the pipeline also works normally (falls back to building).