Skip to content

Add optional voice input with server-side transcription - #91

Closed
en-ver wants to merge 12 commits into
AltanS:mainfrom
en-ver:pi/voice-messages
Closed

Add optional voice input with server-side transcription#91
en-ver wants to merge 12 commits into
AltanS:mainfrom
en-ver:pi/voice-messages

Conversation

@en-ver

@en-ver en-ver commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add an optional, pane-scoped microphone action to eligible empty composers. A foreground recording is transcribed server-side into the ordinary editable, persisted draft; Send remains explicit and nothing is auto-submitted.
  • Make status reporting evidence-based: root-snapshot and pane-read freshness remain independent, and Herdr availability is reported only from fresh root data instead of a reconnecting inference.

Configuration

Voice settings are read at bridge startup. Voice remains disabled while all three dedicated settings are blank:

  • COLLIE_TRANSCRIPTION_API_KEY
  • COLLIE_TRANSCRIPTION_MODEL
  • COLLIE_TRANSCRIPTION_BASE_URL

Any nonblank setting opts in. An omitted model defaults to gpt-4o-transcribe, and an omitted base URL defaults to https://api.openai.com/v1. Official OpenAI requires an API key; a custom HTTP(S) OpenAI-compatible endpoint may be keyless and should include its expected API prefix. A malformed or non-HTTP(S) base URL, or a keyless official OpenAI configuration, leaves voice unavailable. Clear all three settings and restart the plugin to disable voice again.

Of the provider configuration, the browser receives only an enabled/disabled capability. Credentials, model, and endpoint remain server-side. Remote microphone capture requires a secure context such as HTTPS.

Recording and request model

  • Recording is foreground-only. Wake lock is best-effort and held only while actively recording; stopping, cancelling, hiding the page, or changing pane/session releases it and prevents background recording.
  • Recording stops at about five minutes and is rejected above 8 MiB. Collie accepts WebM and MP4, requesting best-effort encoder rates of 24 kb/s and 64 kb/s respectively; browsers may ignore those hints.
  • Voice remains a synchronous completed-file one-shot: one in-memory clip makes one same-origin multipart upload and, after the existing same-origin write/device gate and session/known-pane checks, at most one provider call. Upload and provider redirects are refused, neither network stage retries, and interrupted operations are not resumable.
  • The browser uses a clip-size-derived end-to-end deadline based on a progressing 256 kb/s upload allowance plus provider and scheduling margins. Before multipart parsing, the bridge sets Bun’s nominal 90-second idle allowance; this is neither an exact wall-clock cutoff nor a whole-request deadline. Provider work has an independent 60-second deadline through response-body consumption.
  • At most two known-pane attempts that pass the write gate run concurrently per bridge process; a third receives 429 transcription busy without retry. Decoded provider bodies are capped at 256 KiB and returned transcripts at 8,192 characters.
  • The UI reports Requesting microphone…, Recording, Finishing recording…, and Processing voice…. Finalizing waits for the recorder’s completed file, while processing covers multipart creation, upload, bridge parsing, provider work, and response consumption.

Freshness, privacy, and audit

  • Root-snapshot and pane-read freshness, last-good data, and access failures are tracked independently. Cached root data cannot make a current Herdr-availability claim, and generic loading or pane-local voice work does not alter either freshness domain.
  • Successful text enters the existing pane/session-scoped localStorage draft, remains editable, and is cleared after a verified successful Send or lazily pruned after 48 hours.
  • Server-side Collie does not intentionally persist or log audio, provider response bodies, or transcript contents. Voice requests reaching the handler produce metadata-only outcome and timing audit records; validated MIME, size, and browser-reported recording duration may be included, but content is not.
  • Browser, Bun, OS, and proxy buffering remain outside Collie’s guarantees. The configured provider receives the audio and controls its own retention and logging policy.

Validation

  • Root: 607 passed, 1 skipped; control-script lifecycle passed.
  • Web: 2,270 passed, 10 todo.
  • Root, web, and service-worker typechecks and production builds passed.
  • Tested on an Android device.

@en-ver
en-ver marked this pull request as draft August 11, 2026 15:27
@en-ver
en-ver marked this pull request as ready for review August 11, 2026 16:43
@AltanS

AltanS commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Thanks, this is careful work, but I'm declining it.

Collie isn't a transcription tool. How text gets into the composer is the keyboard's layer, not ours: the composer is a plain textarea and your phone already dictates into it. Owning that layer would mean owning a provider credential, an outbound path from the bridge, and the openai SDK in its deps, none of which Collie should be in the business of.

@AltanS AltanS closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants