Summary
CAI fails to connect to a locally running OpenAI-compatible API gateway that responds correctly to both /v1/models and /v1/chat/completions using standard OpenAI Python and HTTP clients. The gateway serves custom models not in the official OpenAI registry.
Steps to Reproduce
-
Spin up an OpenAI-compatible gateway on localhost (or 127.0.0.1) serving a custom model.
-
Set environment variables:
export OPENAI_API_KEY="test-key"
export OPENAI_API_BASE="http://127.0.0.1/v1"
-
Verify gateway accepts requests:
curl http://127.0.0.1/v1/models returns a model list with "id": "XXXX/XXXX-1.1".curl -X POST http://127.0.0.1/v1/chat/completions ... (using that model) returns completions successfully.- Python OpenAI client with
.chat.completions.create() also works.
-
Start cai and select the custom model name with /model XXXX/XXXX-1.1.
Expected Behavior
CAI should be able to connect and get responses using the custom model, just as the OpenAI Python client and cURL do.
Actual Behavior
CAI fails with:
litellm.APIError: APIError: OpenAIException - Connection error.httpcore.ConnectError: All connection attempts failed- No additional debug output, even with
CAI_LOG_LEVEL=DEBUG or LITELLM_DEBUG=True.
Additional Notes
- The gateway works for both cURL and Python OpenAI client with the same params.
- The custom model appears in
/v1/models and can be POSTed to for completions.
- Model selection in CAI requires manual input (the model doesn’t appear by default).
- Environment is clean, no secrets or product-specific URLs.
Reproducibility
The issue should be reproducible with any OpenAI-compatible API serving a non-OpenAI-official model name.
Note - having the env var OLLAMA_API_BASE seems to resolve this, so it doesn't seem to honour the OPENAI_API_BASE or OPENAI_BASE_URL env vars
Environment
- OS: Linux
- CAI version: 0.5.10
- LiteLLM version in use: 1.68.0
- OpenAI Python client: (v1.75.0+ works for direct calls)
- Gateway: OpenAI-compatible, tested with cURL and OpenAI Python API
Summary
CAI fails to connect to a locally running OpenAI-compatible API gateway that responds correctly to both
/v1/modelsand/v1/chat/completionsusing standard OpenAI Python and HTTP clients. The gateway serves custom models not in the official OpenAI registry.Steps to Reproduce
Spin up an OpenAI-compatible gateway on
localhost(or 127.0.0.1) serving a custom model.Set environment variables:
Verify gateway accepts requests:
curl http://127.0.0.1/v1/modelsreturns a model list with"id": "XXXX/XXXX-1.1".curl -X POST http://127.0.0.1/v1/chat/completions ...(using that model) returns completions successfully..chat.completions.create()also works.Start
caiand select the custom model name with/model XXXX/XXXX-1.1.Expected Behavior
CAI should be able to connect and get responses using the custom model, just as the OpenAI Python client and cURL do.
Actual Behavior
CAI fails with:
litellm.APIError: APIError: OpenAIException - Connection error.httpcore.ConnectError: All connection attempts failedCAI_LOG_LEVEL=DEBUGorLITELLM_DEBUG=True.Additional Notes
/v1/modelsand can be POSTed to for completions.Reproducibility
The issue should be reproducible with any OpenAI-compatible API serving a non-OpenAI-official model name.
Note - having the env var OLLAMA_API_BASE seems to resolve this, so it doesn't seem to honour the OPENAI_API_BASE or OPENAI_BASE_URL env vars
Environment