Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .fallowrc.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"packages/producer/src/**/*.test.ts",
"packages/aws-lambda/src/**/*.test.ts",
"packages/gcp-cloud-run/src/**/*.test.ts",
"packages/gcp-cloud-run/terraform/*.test.ts",
"packages/producer/src/regression-harness.ts",
"packages/producer/src/regression-harness-distributed.test.ts",
"packages/producer/src/regression-harness-lambda-local.ts",
Expand Down Expand Up @@ -403,6 +404,12 @@
// extracting a shared cloud abstraction would couple independent packages.
"packages/aws-lambda/src/handler.ts",
"packages/aws-lambda/src/s3Transport.ts",
// The GCP handler deliberately mirrors the AWS protocol lifecycle while
// retaining provider-specific GCS, HTTP, and Cloud Workflows semantics.
// Its tests also mirror the same wire-contract cases; a cross-provider
// test abstraction would hide the adapter boundary being asserted.
"packages/gcp-cloud-run/src/server.ts",
"packages/gcp-cloud-run/src/server.test.ts",
// sourcePatcher.ts: pre-existing internal clones between the inline-style
// and attribute tag-patchers; only the PatchOperation type gained two
// optional fields here, but the line shift makes fallow re-flag them.
Expand Down
27 changes: 14 additions & 13 deletions bun.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

131 changes: 108 additions & 23 deletions examples/gcp-cloud-run/README.md
Original file line number Diff line number Diff line change
@@ -1,51 +1,136 @@
# Google Cloud Run example

End-to-end deployment + smoke for [`@hyperframes/gcp-cloud-run`](../../packages/gcp-cloud-run) — the Cloud Run + Cloud Workflows adapter for HyperFrames distributed rendering.
End-to-end deployment and parity testing for
[`@hyperframes/gcp-cloud-run`](../../packages/gcp-cloud-run), the Cloud Run +
Cloud Workflows adapter for HyperFrames distributed rendering.

## Layout

```text
scripts/smoke.sh Owner-isolated real-GCP deploy, render, parity, cleanup
sample-events/ v1 and v2 handler request examples
```
scripts/smoke.sh Real-GCP smoke: build → deploy → render → PSNR → destroy
sample-events/ Example request bodies for the Cloud Run handler
(plan.json, render-chunk.json, assemble.json)
```

The Terraform module and the Cloud Workflows definition that the smoke deploys live with the package, at `packages/gcp-cloud-run/terraform/` (including `workflow.yaml`).
The Terraform module and Cloud Workflows definition live in
`packages/gcp-cloud-run/terraform/`.

## Protocol rollout

The workflow defaults to plan protocol v1 when `PlanProtocol` is absent. V2 is
accepted only when the caller explicitly sends `PlanProtocol: "v2"`.

V1 and v2 use disjoint plan locators:

- v1: `PlanGcsUri`
- v2: `PlanV2ManifestGcsUri` and `PlanV2ArtifactGcsPrefix`

The workflow validates that the plan response matches the selected protocol
before starting chunk fan-out. It never silently falls back from v2 to v1.
Deploy the v2 workflow only with a Cloud Run image whose handler implements
the matching v2 request/response contract. An older v1-only handler will keep
serving default v1 requests, but explicit v2 smoke executions will fail closed.

## Prerequisites

- `gcloud` authenticated, with a project that has **billing enabled**
- `terraform` (≥ 1.5), `docker`, `ffmpeg`, `jq` on PATH
- `gcloud` authenticated to a project with billing enabled
- `terraform` (>= 1.5), `ffmpeg`, `ffprobe`, `jq`, `tar`, and `sha256sum`
- the required project APIs already enabled, plus permission to run Cloud
Build and manage Cloud Run, Workflows, GCS, IAM service accounts,
Monitoring, and Artifact Registry resources

## Run the smoke

V1 remains the safe default:

```bash
# Renders the mp4-h264-sdr fixture through the workflow and PSNR-compares it
# against the in-process baseline, then tears the stack down.
./scripts/smoke.sh --project YOUR_GCP_PROJECT --region us-central1
./scripts/smoke.sh \
--project YOUR_GCP_PROJECT \
--region us-central1
```

# Keep the stack up to poke at it:
./scripts/smoke.sh --project YOUR_GCP_PROJECT --keep-stack
Explicitly run v1/v2 end-to-end parity at one or more chunk sizes:

# Render at several chunk sizes to see the fan-out scaling:
./scripts/smoke.sh --project YOUR_GCP_PROJECT --chunk-sizes 30,15,10
```bash
./scripts/smoke.sh \
--project YOUR_GCP_PROJECT \
--region us-central1 \
--protocols v1,v2 \
--chunk-sizes 30,15,10 \
--owner plan-v2-parity
```

Outputs land in `scripts/gcp-smoke-artifacts/`: `results.json`
(`chunkSize × wallClockMs × psnrAvgDb`), the rendered MP4s, and each
workflow execution's describe output.
For each chunk size, parity requires exact equality of:

- decoded RGBA video frames
- decoded 48 kHz stereo PCM audio
- normalized `ffprobe` stream and duration metadata

The encoded MP4 hash and byte count are recorded but are not the equality
oracle because mux metadata can differ without changing decoded output.
Each render is also PSNR-compared with the checked-in in-process fixture
baseline.

## Isolation and cleanup

Every invocation hashes the owner, project, region, and a fresh invocation
nonce into a unique resource prefix such as `hf-smoke-a1b2c3d4e5`. Reusing an
owner label does not reuse old Terraform state or cloud resources. This prefix
stays within GCP service account naming limits. The smoke:

- never uses the static `hyperframes` prefix
- copies the Terraform module into an owner-scoped work directory and uses an
isolated Terraform data directory and state file
- scopes GCS keys, render outputs, the image package/tag, and the default
Artifact Registry repository to that owner
- deletes only an image it built
- deletes the Artifact Registry repository only when that invocation created it
- refuses to enable project APIs, because APIs are shared project state
- stages the bounded Cloud Build source archive in an owner-scoped bucket,
writes build logs to Cloud Logging, and deletes the staging bucket

Cleanup is on by default. It empties and destroys the owner-scoped bucket and
stack, deletes owned image/repository/build-staging resources, then verifies
the Cloud Run service, workflow, buckets, both service accounts, image, and any
test-created repository are absent. Cleanup fails on API or authentication
errors rather than interpreting them as successful deletion. GCP retains the
Cloud Build execution record and Cloud Logging audit entries as project-level
operational history; the smoke test does not attempt to erase audit records.

`--keep-stack` deliberately retains the stack, image, and repository and
prints the exact isolated state directory and Terraform cleanup commands.
Never use it for unattended CI.

Evidence lands under:

```text
scripts/gcp-smoke-artifacts/<owner-hash>/
results.json
parity.json
renders/
terraform/
terraform-data/
```

Use `--image` to test a caller-owned existing image. That image is never
deleted. `--skip-build` requires `--image`; new invocations never inherit an
old invocation's state or image implicitly.

## Test the handler locally

The sample events exercise the same body shape Cloud Workflows sends. With the
container running locally (`PORT=8080`) and credentials that can reach a GCS
bucket, you can drive a single action:
The sample events mirror the request bodies sent by Cloud Workflows:

```bash
# V1
curl -sX POST localhost:8080/ \
-H 'content-type: application/json' \
--data @sample-events/plan.json | jq .

# Explicit v2
curl -sX POST localhost:8080/ \
-H 'content-type: application/json' \
--data @sample-events/plan-v2.json | jq .
```

Replace the `PROJECT` placeholder bucket names and `REPLACE_WITH_PLAN_HASH`
with real values from a prior `plan` response.
Replace `PROJECT`, locator placeholders, and plan hashes with values returned
by the preceding plan action. A complete action sequence is
`plan → renderChunk(s) → assemble`.
14 changes: 14 additions & 0 deletions examples/gcp-cloud-run/sample-events/assemble-v2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"Action": "assemble",
"PlanProtocol": "v2",
"PlanV2ManifestGcsUri": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/v2/manifest.json",
"PlanV2ArtifactGcsPrefix": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/v2/artifacts/sha256",
"PlanHash": "REPLACE_WITH_PLAN_HASH",
"ChunkGcsUris": [
"gs://hyperframes-render-PROJECT/renders/hf-render-demo/chunks/0000.mp4",
"gs://hyperframes-render-PROJECT/renders/hf-render-demo/chunks/0001.mp4"
],
"AudioGcsUri": null,
"OutputGcsUri": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/output.mp4",
"Format": "mp4"
}
1 change: 1 addition & 0 deletions examples/gcp-cloud-run/sample-events/assemble.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"Action": "assemble",
"PlanProtocol": "v1",
"PlanGcsUri": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/plan.tar.gz",
"ChunkGcsUris": [
"gs://hyperframes-render-PROJECT/renders/hf-render-demo/chunks/0000.mp4",
Expand Down
7 changes: 7 additions & 0 deletions examples/gcp-cloud-run/sample-events/plan-v2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"Action": "plan",
"PlanProtocol": "v2",
"ProjectGcsUri": "gs://hyperframes-render-PROJECT/sites/abc123/project.tar.gz",
"PlanOutputGcsPrefix": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/",
"Config": { "fps": 30, "width": 1920, "height": 1080, "format": "mp4" }
}
1 change: 1 addition & 0 deletions examples/gcp-cloud-run/sample-events/plan.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"Action": "plan",
"PlanProtocol": "v1",
"ProjectGcsUri": "gs://hyperframes-render-PROJECT/sites/abc123/project.tar.gz",
"PlanOutputGcsPrefix": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/",
"Config": { "fps": 30, "width": 1920, "height": 1080, "format": "mp4" }
Expand Down
10 changes: 10 additions & 0 deletions examples/gcp-cloud-run/sample-events/render-chunk-v2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
{
"Action": "renderChunk",
"PlanProtocol": "v2",
"PlanV2ManifestGcsUri": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/v2/manifest.json",
"PlanV2ArtifactGcsPrefix": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/v2/artifacts/sha256",
"PlanHash": "REPLACE_WITH_PLAN_HASH",
"ChunkIndex": 0,
"ChunkOutputGcsPrefix": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/",
"Format": "mp4"
}
1 change: 1 addition & 0 deletions examples/gcp-cloud-run/sample-events/render-chunk.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"Action": "renderChunk",
"PlanProtocol": "v1",
"PlanGcsUri": "gs://hyperframes-render-PROJECT/renders/hf-render-demo/plan.tar.gz",
"PlanHash": "REPLACE_WITH_PLAN_HASH",
"ChunkIndex": 0,
Expand Down
Loading
Loading