Skip to content

Batching#17

Merged
chelseacarter29 merged 2 commits intonlweb-ai:mainfrom
Techbert08:batching
Jan 5, 2026
Merged

Batching#17
chelseacarter29 merged 2 commits intonlweb-ai:mainfrom
Techbert08:batching

Conversation

@Techbert08
Copy link
Contributor

This does three things (I love threes):

  1. Update default config to explicitly use 4.1 mini for scoring. This makes it more clear what's in use.
  2. Extending the LLM interface to have a "get_completions" in parallel to "get_completion". The default providers just route "get_completions" to parallel-call "get_completion". But the Pi one instead uses first-class support for parallel calls. There's some yuckiness around lists of kwargs vs kwargs.
  3. I added "ask_llm_parallel" to expose this functionality to the ranker.

Now it works with a local Pi modelserver and with 4.1 mini. My plan is to merge this (assuming it looks good), then finish deploying the updated Pi modelserver to Azure.

@chelseacarter29 chelseacarter29 merged commit 62de835 into nlweb-ai:main Jan 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants