Plan V2 video-frame transport breaks on .hf-complete and clips inactive in a chunk

[JonathanRosado]

Describe the bug

Plan V2's sparse, per-chunk video transport has two linked directory/file-assumption failures:

1. listVideoFramePaths() treats every entry in video-frames/<videoId>/ as a numbered frame. The extraction cache (default-on since perf(engine): extraction cache on by default with atomic publish and LRU gc #1901) puts a zero-byte .hf-complete sentinel in that directory, so planning throws:

   [planV2] cannot derive extracted frame index from .hf-complete
   

2. After excluding that sentinel, Plan V2 correctly assigns no frame artifacts to a chunk where a clip is inactive. Materialization copies files, not empty directories, so video-frames/<inactiveVideoId>/ does not exist. rebuildExtractedFramesFromPlanDir(..., "sparse-v2") nevertheless requires a directory for every video in meta/videos.json.extracted and throws:

   [renderChunk] planDir missing extracted video frames for "video-b":
   /tmp/.../plan/video-frames/video-b not present.
   plan() should have written frames here; the planDir is malformed.
   

This makes valid multi-scene V2 plans fail even though omission of inactive clip frames is the protocol's intended optimization.

Link to reproduction

https://github.com/heygen-com/hyperframes/tree/v0.7.72/packages/producer/src/services/distributed

Steps to reproduce

1. Build a V1 plan with extracted frames and extraction cache enabled, or add an empty .hf-complete beside frame_00001.jpg in the existing planV2.test.ts fixture.
2. Call createPlanV2FromV1(...).
   • Actual: .hf-complete filename parse error.
3. Remove/ignore the sentinel and use two videos with disjoint time ranges so video B is inactive in chunk 0.
4. Materialize { role: "chunk", chunkIndex: 0 }. The manifest correctly selects zero video-frames/B/** artifacts, hence no video-frames/B/ directory exists.
5. Call renderChunk() (or directly rebuildExtractedFramesFromPlanDir(materializedDir, extracted, "sparse-v2")).
   • Actual: missing-directory error for B.

Expected behavior

• .hf-complete is recognized as extraction-cache metadata, excluded from frame dependency selection, and not published as a Plan V2 artifact.
• In sparse-v2 mode, a missing video directory means that video has no frames reachable from this chunk and should be omitted from this chunk's lookup table.
• Dense V1 keeps its current strict missing-directory corruption check.
• Missing artifacts that the manifest says the chunk requires still fail integrity validation.

Actual behavior

The cache sentinel prevents V2 plan publication. Once that is bypassed, any chunk where one declared video is fully inactive fails during sparse materialization/render.

Environment

HyperFrames: 0.7.72
Commit: 72ae39e3d052a185fd0890eccac0085dc9fac60d
Runtime: GCP Cloud Run Gen2, Linux x86_64
Plan protocol: v2
Service shape: 4 vCPU / 16 GiB

Relevant upstream history:

• perf(engine): extraction cache on by default with atomic publish and LRU gc #1901 documents .hf-complete as the extraction cache's atomic-publication sentinel and enables the cache by default.
• feat(producer): add content-addressed plan v2 #2788 introduces exact per-chunk video dependencies, but its fixture has one video active in both chunks and does not cover a fully inactive video.
• feat(gcp-cloud-run): support plan protocol v2 #2790 materializes only chunk-scoped dependencies on GCP.

Suggested narrow fix

• planV2.ts: skip the exact basename .hf-complete while enumerating extracted frame files; assert it is absent from manifest artifacts.
• renderChunk.ts: when indexMode === "sparse-v2", continue if a video output directory is absent; preserve the current throw for dense-v1.
• Add a two-video/disjoint-chunk regression that materializes and rebuilds each sparse target, plus a guard that dense V1 still throws on a missing directory.

This exact scope was validated in a production compatibility image.

Duplicate search

No matching issue or PR was found:

• https://github.com/heygen-com/hyperframes/issues?q=is%3Aissue+%22.hf-complete%22
• https://github.com/heygen-com/hyperframes/issues?q=is%3Aissue+%22missing+extracted+video+frames%22
• https://github.com/heygen-com/hyperframes/issues?q=is%3Aissue+%22sparse-v2%22
• https://github.com/heygen-com/hyperframes/pulls?q=is%3Apr+%22.hf-complete%22

Current main still has both strict assumptions.

[jrusso1020]

Resolved by the two merged fixes that cover the linked failure modes in this issue:

• fix(producer): accept partial color metadata in plan v2 #2814 (58869f0878e304fd39d564b93cc4a8b18e885b4e) handles the extraction-cache .hf-complete sentinel so it is not parsed or published as an extracted frame artifact.
• fix(producer): materialize sparse video directories #2823 (28a340aa6d5ee2333df540632396947414a50347) fixes chunks where a declared video is inactive and therefore has zero selected frame artifacts. The shared Plan v2 materializer now recreates the empty per-video directory declared by integrity-checked meta/videos.json, while still downloading zero unused frame bytes.

The second fix intentionally preserves the strict renderChunk malformed-plan guard instead of treating any absent sparse directory as valid. The regression covers an inactive chunk with zero video artifacts, an active chunk with the expected frame, and unsafe video-ID path rejection. Because the fix is in the shared OSS materializer, it applies to internal S3, AWS Lambda, GCP Cloud Run, and local Plan v2 adapters.

Closing this code issue as completed. Patch release and normal-DEV rollout/replay validation are tracked separately in the rollout plan.