Add AI binding support for pi - #123
Conversation
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
| CF_AI_GATEWAY_API_TOKEN=... | ||
|
|
||
| # To send Workers AI straight to its REST endpoint (no gateway, no cost logs): | ||
| CF_AI_GATEWAY_WAI_DIRECT=true |
There was a problem hiding this comment.
It's unclear to me why we introduced a CF_AI_GATEWAY_WAI_DIRECT env var in the first place, I don't see why we'd want workers-ai traffic to not go through the gateway. I think the CF_AI_GATEWAY_USE_BINDING=false makes a lot more sense, it opts out all the providers from flowing through the binding transport and with CF_AI_GATEWAY_ACCOUNT_ID allows you to use a cross account gateway for all providers
There was a problem hiding this comment.
I think this may have been an artifact of an early misunderstanding where I didn't realize AI Gateway was available via the Workers AI binding. Yeah I think we can remove.
59cd827 to
595d66d
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
|
Looks like upstream PR was merged, shall we update to use that? |
|
Just waiting on a PI release before we can use the upstreamed changes, I'll put up a PR to update once it gets released. The vendored transport changes in this PR are identical to what got merged in to PI, I would like to get this out so we can start battle testing what we upstreamed in to PI. |
|
Drafting this for the time being as I work through on updating the PI implementation to be smarter & more efficient |
595d66d to
24fbea1
Compare
github-actions
Bot
added
kernel
This comment was marked as resolved.
This comment was marked as resolved.
24fbea1 to
0672c92
Compare
This comment was marked as resolved.
This comment was marked as resolved.
9b54ad4 to
f8704a2
Compare
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
f8704a2 to
6f18507
Compare
This comment was marked as resolved.
This comment was marked as resolved.
6f18507 to
976a0e7
Compare
This comment was marked as resolved.
This comment was marked as resolved.
…E_BINDING opt-out
…ays rides the gateway
The WAI knobs existed because Workers AI originally ran on the account-local
WORKERS_AI binding, which can only name same-account gateways; a cross-account
CF_AI_GATEWAY (the internal production shape) therefore needed an escape
hatch. Workers AI has since moved to ordinary gateway routes
(.../{account}/{gateway}/workers-ai/v1), which work cross-account over
HTTPS+token exactly like anthropic/openai -- so the escape hatch no longer
pays for its config surface.
976a0e7 to
00d7293
Compare
|
@Maximo-Guk Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
Preview:
|
Binding-routed gateway traffic went through gateway().run(), the
universal endpoint's [{provider, endpoint, headers, query}] envelope,
which a vendored shim built by splicing the request body into a JSON
string. The gateway also serves its provider-native passthrough over the
AI binding at /ai-gateway/gateways/{gateway}/{provider}/... -- the HTTPS
path minus the account id, since the binding channel carries identity --
and that route accepts exactly the requests pi's API impls already
produce.
So there is nothing left to translate. Binding-routed models take the
binding host as their gateway root and pi's fetch option is the binding's
own fetch, unwrapped; the envelope, the splice, the JSON-object scanner
and the whole vendored ai-gateway-binding-fetch module go away. Method,
headers, query string and the body stream ride through untouched, so
multi-MB prompt bodies are never copied in the isolate.
Google keeps its HTTPS root automatically: `binding` is resolved per
provider, and bindingFor() returns undefined for google (its adapter
can't take a custom fetch). cf-aig-authorization now reaches the gateway,
which recognizes the pre-authentication sentinel and strips it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
bonk break glass |
00d7293 to
945eb5e
Compare
2 participants
Why?
We want to enable users to use AI gateway without needing to provide an API token ( which puts them at risk of credential expiry, rotation, etc ) and makes them need to visit the dashboard just to provision a token during the deploy flow.
How?
Workers AI Gateway can be accessed over the AI binding!
How was this tested?
You can test it with the updated deploy flow which is based off this branches head https://gadgets-deploy.gadgets-staging.workers.dev/deploy ( notice you don't need to provision an api token ), and also locally.