Add VLM-based semantic footage rating tools - #480
Open
abdel-darwish-27 wants to merge 2 commits into
Open
Conversation
Adds four tools that give video libraries semantic understanding with a local vision-language model (Ollama served, e.g. Gemma 4 or Qwen-VL). This covers what static CLIP retrieval cannot: temporal and behavioral semantics. What is included: - vlm_clip_rating: coarse rating of a clip folder (behavior, camera quality, composition, subject visibility, segments, highlights) - vlm_zoom_rating: frame-accurate sub-beat timestamps and deep-dive descriptions for flagged windows - vlm_editorial_ranking: composite scores, leaderboards, and match-cut chains built from the ratings - vlm_comparative_rank: relative ranking of candidate clips with editorial reasoning (shows 4 clips at once for calibrated scores) - vlm-footage-rating skill documenting the workflow - Unit tests for all four tools (Ollama HTTP mocked, no model needed) All stages are idempotent and resume from JSONL output, so re-running after adding footage only processes new clips. Fully local, no API keys.
…el guidance Changes from a portability and model-support audit: - Use the OS temp dir instead of hardcoded /tmp, so the tools work on Windows as well as Linux and macOS. - Add a configurable frame_scale input (320-1280). Smaller values speed up inference and cut VRAM, useful on weak GPUs and with 4b models. - Mark gemma4:12b as the tested and recommended model. Smaller models (gemma3n e2b/e4b, qwen2.5vl 3b/7b) are listed with approximate VRAM but explicitly flagged untested, with guidance on what to expect. - Lower the declared VRAM profile to fit 4GB GPUs (the minimum tier), since the tools support the full model range.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds semantic video understanding to the footage pipeline using a local
vision-language model (Ollama served, e.g. Gemma 4 12B, Gemma 3n, or
Qwen-VL). CLIP tells you what a frame looks like; these tools tell you
what is actually happening in the clip, how well it was shot, and where
the good moments are.
This came out of a real project: rating 101 dog-walk clips for a product
campaign, where static CLIP retrieval got the context right but could not
tell pulling from calm walking, could not score camera stability, and
could not give cut-precise timestamps. The four tools below solve those
gaps, and are generic enough for any footage library.
What is included
Four new tools in
tools/video/, one shared plumbing module, a skill,and tests:
vlm_clip_ratingfocus_promptfor what the edit cares about.vlm_zoom_ratingvlm_editorial_rankingvlm_comparative_rankvlm-footage-ratingskillDesign notes
clips already processed, so re-runs after adding footage only do the new
clips.
non-numeric timestamps are all tolerated) so one bad response never
kills a batch.
up automatically by the registry.
Testing
One pre-existing test is deselected locally because it hangs in the
project's own
video_compose._mux_external_audio(times out at 180s).That file is untouched by this PR. Flagging it in case you want a
separate look.
Example flow
Then the editor (or an agent) can ask "which clips show the product in a
stable close-up?" and get a ranked, timestamped answer.
Model support and testing
The tools were built and validated against Gemma 4 12B (via Ollama,
gemma4:12b, ~8GB VRAM). That is the recommended model and the default.Smaller vision models (gemma3n e2b/e4b, qwen2.5vl 3b/7b) are NOT yet
tested. They are listed with approximate VRAM figures in the skill so
users on smaller GPUs can try them, with the caveat that JSON conformance
and rating quality may differ. The defensive parsing in
vlm_rating_commonis designed to absorb that drift. Contributionsvalidating a smaller model would be very welcome.