Skip to content

add speak detection in om1 - #2676

Open
WenjinFu wants to merge 5 commits into
mainfrom
add-speak-detection
Open

add speak detection in om1#2676
WenjinFu wants to merge 5 commits into
mainfrom
add-speak-detection

Conversation

@WenjinFu

Copy link
Copy Markdown
Contributor

1. OM1 — add-speak-detection

Speaker attribution: OM1 knows who is talking

Every turn, OM1 now tells the model which person in view actually spoke
and when someone says "my name is Sean", the name is saved against that
person, not whoever happens to be most prominent in frame.

How it works

ASR accepts a transcript
  → asr_stream fires POST /speaking {win_start_ms, win_end_ms}   (video-processor, LR-ASD)
  → SpeakerProvider caches the answer
  → runtime waits up to 250ms for it before assembling the prompt
  → prompt: "Speaker: anon_f3b609 (track 73, confidence 0.33) [SPEAKING NOW]"
  → face_memory op:set_name targets that uuid

Changes

File What
internal/providers/speaker.go New. Resolves the active speaker over an audio window; async, TTL'd, single-flight
plugins/inputs/asr/asr_stream.go Starts the resolve on transcript, using the real speech window
internal/providers/face_presence.go Marks the speaking face in the prompt; files memory against them
plugins/actions/face_memory/face_memory.go set_name targets the measured speaker (uuid, track_id fallback)
internal/actions/schema.go BuildSchema honours json:",omitempty"
internal/runtime/runtime.go Waits for the resolve before prompt assembly; logs tool calls
config/*.json5 Prompt sections: who you're talking to, remembering names

Testing

go build ./... and go test ./... pass — 3 new test files, ~950 lines.
(The 3 pre-existing TestConfigFilesMatchSchema failures are unrelated to
this branch.) Verified end-to-end on a G1: person introduces themselves,
gallery and visualization update to their name, including for someone standing
beside the tracked person.

Depends on

OM1-video-processor: /speaking endpoint + profile-face support in LR-ASD.

@WenjinFu
WenjinFu requested review from a team as code owners August 28, 2026 21:11
@github-actions github-actions Bot added tests Test files config Configuration files plugins go core labels Aug 28, 2026
@codecov

codecov Bot commented Aug 28, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

config Configuration files core go plugins tests Test files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants