Fix Granite NAR inference for the current model API; enable NAR on Windows - #45
Closed
sumgup0 wants to merge 3 commits into
Closed
Fix Granite NAR inference for the current model API; enable NAR on Windows#45sumgup0 wants to merge 3 commits into
sumgup0 wants to merge 3 commits into
Conversation
…ndows The published ibm-granite/granite-speech-4.1-2b-nar custom code drifted: its inference entrypoint was renamed generate() -> transcribe() (can_generate() now returns False), and the output changed from decoded strings (output.text_preds) to raw token-id tensors (output.preds). The NAR backend still called the old API, so transcribe() raised AttributeError on the current revision under transformers 5.x -- on every platform, not just Windows. The catalog gate merely kept Linux/Mac users from discovering it; the friend's pinned Linux NAR works only because it predates the upstream rename. - asr.py: make NAR inference version-robust. Prefer model.transcribe() and fall back to model.generate(); prefer output.text_preds and else decode output.preds token ids with a tokenizer now loaded in _load(). Old and new model revisions both work, so Linux/Mac do not regress. - catalog.py / profiles.py: lift the deliberate "NAR unsupported on Windows" gate now that NAR runs on Windows CUDA. AR (granite-speech-4.1-2b) stays the Windows default; NAR becomes selectable. Incremental transcription stays gated on Windows (separate validation) with an accurate reason. Verified on an RTX 5090 (Blackwell, cu128): NAR loads on cuda:0, warm transcribe ~0.06s (vs ~0.33s for AR), correct transcript. New unit tests cover both the new (transcribe/preds) and old (generate/text_preds) code paths without needing the model downloaded. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the nar-current-api review: - models/prefetch.py: _prefetch_granite_nar now also caches the tokenizer (AutoTokenizer) so a fresh prefetch covers everything the NAR backend loads with local_files_only=True; it previously relied on AutoProcessor co-caching the tokenizer files. - asr.py: _nar_decode uses truthiness instead of `is not None`, so a present-but-empty text_preds/preds falls through instead of raising IndexError; documented that already-decoded text_preds wins when both are present. New tests pin both behaviours. - README/CONTEXT: Granite NAR is now selectable on Windows CUDA (AR stays the default); updated the support matrix and the Windows ASR list. ROCm remains unsupported on Windows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Upstream PR paulbrav#43 added a new `windows_openvino` runtime profile that set `granite_nar_unsupported_reason=GRANITE_NAR_UNSUPPORTED_WINDOWS`. This branch deletes that constant when lifting the NAR-on-Windows gate, so git's textual auto-merge (the two changes never overlapped a line) left a dangling reference that `ty` flagged as unresolved-reference. Lift the gate on windows_openvino too, consistent with NAR being selectable on every Windows profile -- device resolution handles actual CUDA availability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
paulbrav
added a commit
that referenced
this pull request
Jun 14, 2026
Owner
|
Superseded by #47, which merged your work (thanks @sumgup0!) rebased onto current master + review follow-ups: the NAR tokenizer is now loaded lazily so the Linux gfx1151 startup path is unchanged, plus a clearer inference error. The version-robust transcribe()/preds fix is in master as of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The published
ibm-granite/granite-speech-4.1-2b-narremote code (loaded viatrust_remote_code) has drifted: its inference entrypoint was renamedmodel.generate()->model.transcribe()(can_generate()now returnsFalse),and its output changed from decoded strings (
output.text_preds) to raw token-idtensors (
output.preds). The NAR backend still called the old API, sotranscribe_waveform()raisesAttributeErroron the current model revisionunder transformers 5.x - on every platform, not just Windows. The catalog gate
merely kept Linux/macOS users from discovering it.
This makes the NAR inference call version-robust (works against both the old and
new model revisions, protecting the pinned Linux NAR default) and lifts the
deliberate "NAR unsupported on Windows" gate now that NAR runs on Windows CUDA.
Changes
asr.py- version-robust NAR inference. Prefermodel.transcribe(), fallback to
model.generate(); preferoutput.text_preds, else decodeoutput.predstoken ids with a tokenizer now loaded in_load(). Old and newmodel revisions both work, so the Linux NAR default does not regress.
catalog.py/profiles.py- lift the Windows gate. AddWindowsto theNAR entry's
supported_platformsand dropgranite_nar_unsupported_reasonfromthe Windows profiles (including the new
windows_openvinoone). Granite AR staysthe Windows default; NAR becomes selectable. Incremental transcription stays
gated on Windows (separate validation).
prefetch.py-_prefetch_granite_naralso caches the tokenizer, so a freshmodels prefetchcovers everything_load(local_files_only=True)needs.transcribe/preds) and old (generate/text_preds)code paths; inverted the policy tests that asserted NAR-rejected-on-Windows;
README/CONTEXT updated.
Why this likely matters on Linux too
The Linux NAR default runs the same
transcribe_waveform. It works today onlybecause the cached model revision predates the upstream
generate->transcriberename; the next time that remote code refreshes (or transformers is bumped), the
Linux default would hit the same
AttributeError. The version-robust fix coversboth.
Testing
cuda:0, ~0.06 s warmtranscribe (vs ~0.33 s for Granite AR), correct transcript via the production
path with
local_files_only=True._nar_infer/_nar_decode(both API shapes) pass withoutthe model downloaded.
ruff check+tyclean. Coexists with the OpenVINO backend - thetylanecaught the
windows_openvinoprofile interaction during the rebase onto master.