Plan V2 rejects valid video metadata when ffprobe reports only some color fields

[JonathanRosado]

Describe the bug

Plan Protocol V2 cannot plan an otherwise valid SDR video when ffprobe reports one color field but omits the others. The engine intentionally normalizes absent components to empty strings, but planV2.ts validates all three components with the generic non-empty-string reader.

Observed error:

[planV2] meta/videos.json.extracted[3].metadata.colorTransfer must be a non-empty string

Plan V1/local rendering accepts the same source. This is a metadata-validation mismatch only; the media decodes and renders correctly.

Link to reproduction

https://github.com/heygen-com/hyperframes/tree/v0.7.72/packages/producer

Steps to reproduce

Create an H.264 file whose stream declares the matrix but not transfer/primaries:

ffmpeg -f lavfi -i 'color=c=gray:s=320x180:r=24:d=1' \
  -c:v libx264 -pix_fmt yuv420p -colorspace bt709 -an partial-color.mp4

ffprobe -v error -select_streams v:0 \
  -show_entries stream=codec_name,pix_fmt,color_space,color_transfer,color_primaries \
  -of json partial-color.mp4

The stream is valid. ffprobe returns color_space: "bt709" while omitting color_transfer and color_primaries.

Use the file in a <video> and run distributed planning with PlanProtocol: "v2" (or create a V1 plan and call createPlanV2FromV1). The engine writes:

"colorSpace": {
  "colorTransfer": "",
  "colorPrimaries": "",
  "colorSpace": "bt709"
}

createPlanV2FromV1 then throws because readVideoMetadata() calls readString() for every component.

The source-level contradiction is visible at v0.7.72: the engine's ffprobe adapter uses videoStream.color_transfer || "", while the Plan V2 reader requires the field to be non-empty.

Expected behavior

V2 accepts each color component as a string, including "" for an ffprobe-omitted component, while retaining null when no color metadata exists at all. Output behavior should match V1.

Actual behavior

Planning aborts before publication with PlanV2IntegrityError, even though the file is valid SDR and decodable.

Environment

HyperFrames: 0.7.72
Commit: 72ae39e3d052a185fd0890eccac0085dc9fac60d
Runtime: GCP Cloud Run Gen2, Linux x86_64
Node image: node:22-bookworm-slim
Service shape: 4 vCPU / 16 GiB
Real input: H.264, yuv420p, 1920x1080, 24 CFR

The production failure occurred with ffprobe reporting color_space=bt709 and empty transfer/primaries. No decode or pixel-path change was needed to render the file successfully.

Suggested narrow fix

In packages/producer/src/services/distributed/planV2.ts, use a string-type validator that permits "" only for colorTransfer, colorPrimaries, and colorSpace; keep the generic non-empty validator everywhere else. Add a regression with partial ffprobe metadata.

We validated that scope in a compatibility image: validator-only, no color conversion, decode, or pixel-path changes.

Duplicate search

No matching issue or PR was found:

• https://github.com/heygen-com/hyperframes/issues?q=is%3Aissue+colorTransfer
• https://github.com/heygen-com/hyperframes/issues?q=is%3Aissue+colorPrimaries
• https://github.com/heygen-com/hyperframes/issues?q=is%3Aissue+PlanV2IntegrityError
• https://github.com/heygen-com/hyperframes/pulls?q=is%3Apr+colorTransfer

The Plan V2 PR chain, including #2788, contains no fix for this validation mismatch.

[jrusso1020]

Closed by #2814 (merge commit 58869f0878e3).

Root cause

This was a real Plan v2 compatibility bug, not invalid user media. The engine intentionally emits a non-null color object when any ffprobe color component exists and normalizes omitted components to empty strings. Plan v1 accepted that contract, but Plan v2 parsed all three optional color components with the generic non-empty-string validator, so valid partial metadata such as color_space=bt709 with omitted transfer/primaries failed during planning.

Live validation also exposed an adjacent Plan v2 issue: the engine-owned zero-byte .hf-complete extraction-cache sentinel could be treated as a frame artifact. The original handling was tied to exact-dependency metadata and therefore did not protect the metadata-absent full-source fallback.

Fix

• Permit empty strings only for the three optional color components while preserving strict validation everywhere else.
• Validate the canonical video-frames/<videoId>/.hf-complete sentinel as a zero-byte regular file in every dependency mode.
• Omit only that canonical sentinel from publication; malformed file/directory/symlink forms fail closed, while unknown nested paths remain conservatively included.
• Add partial-color, metadata-absent, exact/full-source, malformed-sentinel, and render regression coverage.

Validation

Producer unit tests, engine ffprobe tests, typechecks, Windows and full regression CI passed. Live owner-isolated GCP v1/v2 testing produced matching decoded frames, audio, and normalized metadata, with byte-identical outputs at 46.85 dB PSNR.