What's Next for LLM Providers Settings?

[dhbrojas]

Note

I'm not super looped into the development plan for LLM Providers in Zed. Hence, some elements of this post may have been discussed elsewhere already. If this is the case, I apologise. I'm also not sure whether this thread best belongs in the "LLMs and Zed Agent" category or the broader "Zed Development" category.

Background

The LLM Providers section of the Agent Panel, lets you configure the inference provider (infrastructure) and language model (weights) to use in your conversations with the agent.

Up until recently, only a small set of pre-defined inference providers and language models were available (e.g. OpenAI, Anthropic, xAI). Each provider had to be explicitly defined in the code as a standalone Rust crate and registered in the language_models crate.

At the moment, OpenRouter (a universal interface for LLM inference) is the preferred way of using models from providers not implemented in Zed.

Recently, a new OpenAI compatible provider abstraction was added. It leverages the fact that most providers use the same API format (the OpenAI format). This lets users use any inference provider and language model pair they desire as long as they configure the necessary information (e.g. API URL, API key, model slug, maximum model prompt and completions tokens, etc.) in the Zed UI or in the Zed JSON settings.

Problem Statement

First of all, since the addition of the OpenAI compatible provider, it doesn't make sense to systematically create a new crate for providers that comply with the OpenAI format. On the other hand, it would be burdensome to let users manually configure a new OpenAI compatible LLM provider from scratch as it requires a bit of knowledge of the API and because the configuration must be updated for every time a model is released or deprecated. This is especially true for popular providers.

Furthermore, every few weeks a new provider comes along (at the moment there are two open PRs to add support for the Cerebras and Z.ai provider). Interestingly, some providers introduce new models (such as the Z.ai provider which provides direct access to the models Z.ai develops) while other providers serve existing models (such as the Cerebras provider which serves Qwen at a much higher throughput). This growing diversity is hard to tackle from a user experience perspective. You don't want users struggling with configuration that must be kept up to date but you also want a small code/implementation footprint.

It's also unclear the role OpenRouter can play here. OpenRouter provides a convenient unified interface for LLMs and is already used in Zed. However, it's a little hard to configure given the sheer number of available models, providers, etc.

Discussion

I'm interested in the opinion of the developers and community on this matter. There are few possible avenues to investigate. I'm sure the community will have great suggestions on top of the two below:

Greater OpenRouter Integration

The abstraction OpenRouter proposes seems to solve the problem discussed in this post. However, at the moment, one cannot change the list of models available through OpenRouter nor the provider. The experience of using OpenRouter on Zed is slightly worse than that of established providers such as OpenAI or Anthropic.

Should OpenRouter be seen as more than just a LLM provider? Perhaps, it can have its own dedicated section in the LLM Providers configuration panel to allow for more fine-grained control.

Related: #35394

LLM API Formats as Code, LLM Providers as Configuration

While there exists dozens of providers and thousands of models, there are only 3-4 widely-used API formats.

Hence, instead of having each provider implement its own crate, we could add support for these 3-4 API formats (OpenAI, Anthropic, Google, etc.) and having all providers be registered as JSON, in the same way the OpenAI compatible providers are configured now.

Zed would then come prepackaged with popular providers (e.g. OpenAI, xAI, DeepSeek, Qwen, Z.ai, etc.) also allowing users to configure their own.

---

Happy to hear your thoughts.

[kinoute]

Being able to reorganize the LLM providers order would be nice (could not find how to do it). Right now Recommended/Zed is always on top but if you use other LLM providers, it is kinda of annoying.