feat(capture): vertex service-account auth for vision captioning - #3362
Open
WaterrrForever wants to merge 2 commits into
Open
feat(capture): vertex service-account auth for vision captioning#3362WaterrrForever wants to merge 2 commits into
WaterrrForever wants to merge 2 commits into
Conversation
The Vertex Gemini endpoint rejects plain API keys, so tenants whose Google access is service-account based could never enable vision captioning: the vision phase degraded with provider-error and asset-descriptions.md fell back to filename echoes. Opt in with GOOGLE_GENAI_USE_VERTEXAI=true (the @google/genai SDK convention). Credentials come from GOOGLE_SERVICE_ACCOUNT_INFO as inline SA JSON when set, else ADC; project falls back to the SA's own project_id and location defaults to global. The flag outranks a bare GEMINI_API_KEY because a tenant that sets it is saying its key material is Vertex-side; OPENROUTER_API_KEY stays the top-priority opt-in. Vertex default model is gemini-2.5-flash-lite — Vertex does not publish the 3.x flash-lite preview name the Gemini API path defaults to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vertex serves the GA gemini-3.1-flash-lite name (the -preview alias is Gemini-API-only), so the Vertex path can default to the same tier as the API-key path instead of stepping down to 2.5. Verified multimodal 200 on the worker service account.
WaterrrForever
force-pushed
the
miao/capture-vertex-gemini
branch
from
August 20, 2026 09:29
0c1c259 to
80b536d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The capture pipeline's vision captioning only accepts a Gemini API key (
GEMINI_API_KEY/GOOGLE_API_KEY). Tenants whose Google access is service-account based — HeyGen's own workers included — can never turn it on: the Vertex endpoint rejects plain API keys, and the GCP key present in the worker env is not valid for the Gemini API. Observed effect: every capture'svisionphase degrades withprovider-errorandasset-descriptions.mdfalls back to filename echoes, which downstream agents then compensate for by opening images one by one.What
GOOGLE_GENAI_USE_VERTEXAI=true(the@google/genaiSDK's own convention) routes captioning through Gemini on Vertex AI:GOOGLE_SERVICE_ACCOUNT_INFO(inline SA JSON — the same variable HeyGen's multivendor Gemini provider already reads), else ADCproject_id; location defaults toglobal(GOOGLE_CLOUD_PROJECT/GOOGLE_CLOUD_LOCATIONoverride)OPENROUTER_API_KEY→ Vertex flag →GEMINI_API_KEY. The flag outranks a bare key because a tenant that sets it is saying its key material is Vertex-side.gemini-2.5-flash-lite(Vertex does not publish thegemini-3.1-flash-lite-previewname the API-key path defaults to; verified 404 vs 200).HYPERFRAMES_GEMINI_MODELstill overrides.asset-descriptions.mdheader now reflects the Vertex option.Testing
vitest run src/capture/contentExtractor.test.ts— 18 passed (5 new: config resolution × 4, provider selection through the mocked SDK asserting the Vertex default model).tsc --noEmit— no errors in the touched files.GOOGLE_GENAI_USE_VERTEXAI=true+ the existing worker SA:visionphase completed,3 images captioned with Vertex Gemini, captions are genuine content descriptions (og-image: "A black text logo, ANTHROPIC, is centered on a light background"), 32 SVGs rasterized+captioned.🤖 Generated with Claude Code