Vision Proxy fails with Server error 500 when using VS Code language model customendpoint, while direct image chat works

[JohnyWu]

Summary

When using DeepSeek V4 Flash with an image attachment, the DeepSeek Vision Proxy fails with:

[UNKNOWN] Vision proxy request failed (unknown).

The selected vision proxy model is a VS Code language model:

Azure Foundry GPT-5.5 via LiteLLM (azure/gpt-5.5) - customendpoint

However, the same model can process the same image successfully when used directly in VS Code Chat. So the model itself supports image input, but the Vision Proxy call from this extension fails.

Environment

• Extension: DeepSeek V4 for Copilot Chat
• Extension version: 0.7.0
• Publisher: Vizards
• VS Code version: 1.132.0
• OS: macOS
• Architecture: arm64
• DeepSeek model: deepseek-v4-flash
• Vision proxy source: VS Code language model
• Vision model: azure/gpt-5.5
• Vision model vendor: customendpoint
• Vision model name: Azure Foundry GPT-5.5 via LiteLLM

Steps to Reproduce

1. Install and enable DeepSeek V4 for Copilot Chat.
2. Select deepseek-v4-flash in Copilot Chat.
3. Open DeepSeek Vision Proxy.
4. Set Vision proxy source to VS Code model.
5. Select Azure Foundry GPT-5.5 via LiteLLM (azure/gpt-5.5) - customendpoint.
6. Send a chat message to DeepSeek with an attached image.
7. The request fails with [UNKNOWN] Vision proxy request failed (unknown).

Expected Behavior

The extension should call the selected VS Code vision-capable language model, convert the image to text, and pass the description to DeepSeek.

Actual Behavior

The Vision Proxy fails with an unknown error. The extension log shows a server-side 500 error from the VS Code/Copilot language model call.

Important Observation

The same image works when sent directly to the selected VS Code language model in VS Code Chat.

Direct use:

• Model: Azure Foundry GPT-5.5 via LiteLLM
• Image input: works
• The model correctly describes the screenshot

But through this extension's Vision Proxy:

• Same model
• Same image
• Fails with Server error: 500

This suggests the model supports image input, but there may be a compatibility issue in how the extension calls the VS Code Language Model API for image attachments, possibly around message format, image encoding, request options, or customendpoint/LiteLLM compatibility.

Logs

2026-08-08 13:26:55.502 [info] Vision proxy: azure/gpt-5.5 selected=id="azure/gpt-5.5" vendor="customendpoint" name="Azure Foundry GPT-5.5 via LiteLLM" family="azure/gpt-5.5" version="1.0.0" targetChatSessionType=n/a

2026-08-08 13:27:12.564 [error] Vision proxy error: kind=unknown message="Server error: 500"
Error: Server error: 500
    at GG._provideLanguageModelResponse (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1690:14392)
    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
    at async GG.provideLanguageModelResponse (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/copilot/dist/extension.js:1690:15505)

[JohnyWu]