Skip to content

Add VLM-based semantic footage rating tools - #480

Open
abdel-darwish-27 wants to merge 2 commits into
calesthio:mainfrom
abdel-darwish-27:feat/vlm-clip-rating
Open

Add VLM-based semantic footage rating tools#480
abdel-darwish-27 wants to merge 2 commits into
calesthio:mainfrom
abdel-darwish-27:feat/vlm-clip-rating

Conversation

@abdel-darwish-27

@abdel-darwish-27 abdel-darwish-27 commented Aug 8, 2026

Copy link
Copy Markdown

Summary

Adds semantic video understanding to the footage pipeline using a local
vision-language model (Ollama served, e.g. Gemma 4 12B, Gemma 3n, or
Qwen-VL). CLIP tells you what a frame looks like; these tools tell you
what is actually happening in the clip, how well it was shot, and where
the good moments are.

This came out of a real project: rating 101 dog-walk clips for a product
campaign, where static CLIP retrieval got the context right but could not
tell pulling from calm walking, could not score camera stability, and
could not give cut-precise timestamps. The four tools below solve those
gaps, and are generic enough for any footage library.

What is included

Four new tools in tools/video/, one shared plumbing module, a skill,
and tests:

Tool What it does
vlm_clip_rating Coarse pass over a clip folder: behavior, energy, camera stability, shot type, composition, subject (product) visibility, timestamped segments and highlights. Configurable focus_prompt for what the edit cares about.
vlm_zoom_rating Re-examines flagged windows at 4 fps and produces frame-accurate sub-beats: precise start/end seconds, camera angle, subject facing direction (for match cuts), deep-dive descriptions, and vibe.
vlm_editorial_ranking Builds composite scores (stability, quality, subject, composition, vibe with tunable weights), per-purpose leaderboards, best-moment extraction, and match-cut chains.
vlm_comparative_rank Optional second-opinion pass: shows 4 candidate clips in one context, asks for a relative ranking, calibrated scores, and reasons for best/worst. Use to break ties.
vlm-footage-rating skill Documents the workflow, schemas, and editing recipes.
Tests 29 unit tests covering all four tools. Ollama HTTP calls are mocked, frame extraction uses synthetic ffmpeg clips, so the suite runs with no model, no network, no GPU.

Design notes

  • Fully local: requires only ffmpeg and an Ollama vision model. No API keys.
  • Every stage is idempotent: it reads the existing JSONL output and skips
    clips already processed, so re-runs after adding footage only do the new
    clips.
  • VLM output is parsed defensively (prose-wrapped JSON, field-name drift,
    non-numeric timestamps are all tolerated) so one bad response never
    kills a batch.
  • The tools register through the standard BaseTool pattern and are picked
    up automatically by the registry.

Testing

make lint        # passes (py_compile on all targets + new files)
make test        # full suite: 991 passed, 11 skipped

One pre-existing test is deselected locally because it hangs in the
project's own video_compose._mux_external_audio (times out at 180s).
That file is untouched by this PR. Flagging it in case you want a
separate look.

Example flow

from tools.video.vlm_clip_rating import VlmClipRating
from tools.video.vlm_zoom_rating import VlmZoomRating
from tools.video.vlm_editorial_ranking import VlmEditorialRanking

VlmClipRating().execute({
    "input_dir": "/path/to/clips",
    "output_path": "/path/to/clip_tags.jsonl",
    "focus_prompt": "a black collar (the product)",
})
VlmZoomRating().execute({
    "ratings_path": "/path/to/clip_tags.jsonl",
    "output_path": "/path/to/clip_zooms.jsonl",
})
VlmEditorialRanking().execute({
    "ratings_path": "/path/to/clip_tags.jsonl",
    "zooms_path": "/path/to/clip_zooms.jsonl",
    "output_path": "/path/to/editorial_rankings.json",
})

Then the editor (or an agent) can ask "which clips show the product in a
stable close-up?" and get a ranked, timestamped answer.

Model support and testing

The tools were built and validated against Gemma 4 12B (via Ollama,
gemma4:12b, ~8GB VRAM). That is the recommended model and the default.

Smaller vision models (gemma3n e2b/e4b, qwen2.5vl 3b/7b) are NOT yet
tested. They are listed with approximate VRAM figures in the skill so
users on smaller GPUs can try them, with the caveat that JSON conformance
and rating quality may differ. The defensive parsing in
vlm_rating_common is designed to absorb that drift. Contributions
validating a smaller model would be very welcome.

Adds four tools that give video libraries semantic understanding with a
local vision-language model (Ollama served, e.g. Gemma 4 or Qwen-VL).
This covers what static CLIP retrieval cannot: temporal and behavioral
semantics.

What is included:
- vlm_clip_rating: coarse rating of a clip folder (behavior, camera
  quality, composition, subject visibility, segments, highlights)
- vlm_zoom_rating: frame-accurate sub-beat timestamps and deep-dive
  descriptions for flagged windows
- vlm_editorial_ranking: composite scores, leaderboards, and match-cut
  chains built from the ratings
- vlm_comparative_rank: relative ranking of candidate clips with
  editorial reasoning (shows 4 clips at once for calibrated scores)
- vlm-footage-rating skill documenting the workflow
- Unit tests for all four tools (Ollama HTTP mocked, no model needed)

All stages are idempotent and resume from JSONL output, so re-running
after adding footage only processes new clips. Fully local, no API keys.
…el guidance

Changes from a portability and model-support audit:
- Use the OS temp dir instead of hardcoded /tmp, so the tools work on
  Windows as well as Linux and macOS.
- Add a configurable frame_scale input (320-1280). Smaller values speed
  up inference and cut VRAM, useful on weak GPUs and with 4b models.
- Mark gemma4:12b as the tested and recommended model. Smaller models
  (gemma3n e2b/e4b, qwen2.5vl 3b/7b) are listed with approximate VRAM
  but explicitly flagged untested, with guidance on what to expect.
- Lower the declared VRAM profile to fit 4GB GPUs (the minimum tier),
  since the tools support the full model range.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant