diff --git a/docs/customize/deep-dives/model-capabilities.mdx b/docs/customize/deep-dives/model-capabilities.mdx new file mode 100644 index 0000000000..df1daefc55 --- /dev/null +++ b/docs/customize/deep-dives/model-capabilities.mdx @@ -0,0 +1,246 @@ +--- +title: Model Capabilities +description: Understanding and configuring model capabilities for tools and image support +keywords: [capabilities, tools, function calling, image input, config] +--- + +Continue needs to know what features your models support to provide the best experience. This guide explains how model capabilities work and how to configure them. + +## What are Model Capabilities? + +Model capabilities tell Continue what features a model supports: + +- **`tool_use`** - Whether the model can use tools and functions +- **`image_input`** - Whether the model can process images + +Without proper capability configuration, you may encounter issues like: + +- Agent mode being unavailable (requires tools) +- Tools not working at all +- Image uploads being disabled + +## How Continue Detects Capabilities + +Continue uses a two-tier system for determining model capabilities: + +### 1. Automatic Detection (Default) + +Continue automatically detects capabilities based on your provider and model name. For example: + +- **OpenAI**: GPT-4 and GPT-3.5 Turbo models support tools +- **Anthropic**: Claude 3.5+ models support both tools and images +- **Ollama**: Most models support tools, vision models support images +- **Google**: All Gemini models support function calling + +This works well for popular models, but may not cover custom deployments or newer models. + +For implementation details, see: + +- [toolSupport.ts](https://github.com/continuedev/continue/blob/main/core/llm/toolSupport.ts) - Tool capability detection logic +- [@continuedev/llm-info](https://www.npmjs.com/package/@continuedev/llm-info) - Image support detection + +### 2. Manual Configuration + +You can add capabilities to models that Continue doesn't automatically detect in your `config.yaml`. + + +You cannot override autodetection - you can only add capabilities. Continue will always use its built-in knowledge about your model in addition to any capabilities you specify. + + +```yaml +models: + - name: my-custom-gpt4 + provider: openai + apiBase: https://my-deployment.com/v1 + model: gpt-4-custom + capabilities: + - tool_use + - image_input +``` + +## When to Add Capabilities + +Add capabilities when: + +1. **Using custom deployments** - Your API endpoint serves a model with different capabilities than the standard version +2. **Using newer models** - Continue doesn't yet recognize a newly released model +3. **Experiencing issues** - Autodetection isn't working correctly for your setup +4. **Using proxy services** - Some proxy services modify model capabilities + +## Configuration Examples + +### Basic Configuration + +Add tool support for a model that Continue doesn't recognize: + +```yaml +models: + - name: custom-model + provider: openai + model: my-fine-tuned-gpt4 + capabilities: + - tool_use +``` + + + The `tool_use` capability is for native tool/function calling support. The + model must actually support tools for this to work. + + + + **Experimental**: System message tools are available as an experimental feature + for models without native tool support. These are not automatically used as a + fallback and must be explicitly configured. Most models are trained for native + tools, so system message tools may not work as well. + + +### Disable Capabilities + +Explicitly set no capabilities (autodetection will still apply): + +```yaml +models: + - name: limited-claude + provider: anthropic + model: claude-4.0-sonnet + capabilities: [] # Empty array doesn't disable autodetection +``` + + + An empty capabilities array does not disable autodetection. Continue will + still detect and use the model's actual capabilities. To truly limit a model's + capabilities, you would need to use a model that doesn't support those + features. + + +### Multiple Capabilities + +Enable both tools and image support: + +```yaml +models: + - name: multimodal-gpt + provider: openai + model: gpt-4-vision-preview + capabilities: + - tool_use + - image_input +``` + +## Common Scenarios + +Some providers and custom deployments may require explicit capability configuration: + +- **OpenRouter**: May not preserve the original model's capabilities +- **Custom API endpoints**: May have different capabilities than standard models +- **Local models**: May need explicit capabilities if using non-standard model names + +Example configuration: + +```yaml +models: + - name: custom-deployment + provider: openai + apiBase: https://custom-api.company.com/v1 + model: custom-gpt + capabilities: + - tool_use # If supports function calling + - image_input # If supports vision +``` + +## Troubleshooting + +For troubleshooting capability-related issues like Agent mode being unavailable or tools not working, see the [Troubleshooting guide](/troubleshooting#agent-mode-is-unavailable-or-tools-arent-working). + +## Best Practices + +1. **Start with autodetection** - Only override if you experience issues +2. **Test after changes** - Verify tools and images work as expected +3. **Keep Continue updated** - Newer versions improve autodetection + +Remember: Setting capabilities only adds to autodetection. Continue will still use its built-in knowledge about your model in addition to your specified capabilities. + +## Model Capability Support + +This matrix shows which models support tool use and image input capabilities. Continue auto-detects these capabilities, but you can override them if needed. + +### OpenAI + +| Model | Tool Use | Image Input | Context Window | +| :------------ | -------- | ----------- | -------------- | +| o3 | Yes | No | 128k | +| o3-mini | Yes | No | 128k | +| GPT-4o | Yes | Yes | 128k | +| GPT-4 Turbo | Yes | Yes | 128k | +| GPT-4 | Yes | No | 8k | +| GPT-3.5 Turbo | Yes | No | 16k | + +### Anthropic + +| Model | Tool Use | Image Input | Context Window | +| :---------------- | -------- | ----------- | -------------- | +| Claude 4 Sonnet | Yes | Yes | 200k | +| Claude 3.5 Sonnet | Yes | Yes | 200k | +| Claude 3.5 Haiku | Yes | Yes | 200k | + +### Google + +| Model | Tool Use | Image Input | Context Window | +| :--------------- | -------- | ----------- | -------------- | +| Gemini 2.5 Pro | Yes | Yes | 2M | +| Gemini 2.0 Flash | Yes | Yes | 1M | + +### Mistral + +| Model | Tool Use | Image Input | Context Window | +| :-------------- | -------- | ----------- | -------------- | +| Devstral Medium | Yes | No | 32k | +| Mistral | Yes | No | 32k | + +### DeepSeek + +| Model | Tool Use | Image Input | Context Window | +| :---------------- | -------- | ----------- | -------------- | +| DeepSeek V3 | Yes | No | 128k | +| DeepSeek Coder V2 | Yes | No | 128k | +| DeepSeek Chat | Yes | No | 64k | + +### xAI + +| Model | Tool Use | Image Input | Context Window | +| :----- | -------- | ----------- | -------------- | +| Grok 4 | Yes | Yes | 128k | + +### Moonshot AI + +| Model | Tool Use | Image Input | Context Window | +| :------ | -------- | ----------- | -------------- | +| Kimi K2 | Yes | Yes | 128k | + +### Qwen + +| Model | Tool Use | Image Input | Context Window | +| :---------------- | -------- | ----------- | -------------- | +| Qwen Coder 3 480B | Yes | No | 128k | + +### Ollama (Local Models) + +| Model | Tool Use | Image Input | Context Window | +| :------------- | -------- | ----------- | -------------- | +| Qwen 3 Coder | Yes | No | 32k | +| Devstral Small | Yes | No | 32k | +| Llama 3.1 | Yes | No | 128k | +| Llama 3 | Yes | No | 8k | +| Mistral | Yes | No | 32k | +| Codestral | Yes | No | 32k | +| Gemma 3 4B | Yes | No | 8k | + +### Notes + +- **Tool Use**: Function calling support (tools are required for Agent mode) +- **Image Input**: Processing images +- **Context Window**: Maximum number of tokens the model can process in a single request + +--- + +**Is your model missing or incorrect?** Help improve this documentation! You can edit this page on GitHub using the link below. diff --git a/docs/customize/model-providers/more/openrouter.mdx b/docs/customize/model-providers/more/openrouter.mdx index 67f1c60511..f1be540173 100644 --- a/docs/customize/model-providers/more/openrouter.mdx +++ b/docs/customize/model-providers/more/openrouter.mdx @@ -71,3 +71,50 @@ For example, to prevent extra long prompts from being compressed, you can explic Learn more about available settings [here](https://openrouter.ai/docs). + +## Model Capabilities + +OpenRouter models may require explicit capability configuration because the proxy doesn't always preserve the original model's function calling support. + + + Continue automatically uses system message tools for models that don't support native function calling, so Agent mode should work even without explicit capability configuration. However, you can still override capabilities if needed. + + +If you're experiencing issues with Agent mode or tools not working, you can add the capabilities field: + + + + ```yaml title="config.yaml" + models: + - name: Claude via OpenRouter + provider: openrouter + model: anthropic/claude-3.5-sonnet + apiBase: https://openrouter.ai/api/v1 + apiKey: + capabilities: + - tool_use # Enable function calling for Agent mode + - image_input # Enable image upload support + ``` + + + ```json title="config.json" + { + "models": [ + { + "title": "Claude via OpenRouter", + "provider": "openrouter", + "model": "anthropic/claude-3.5-sonnet", + "apiBase": "https://openrouter.ai/api/v1", + "apiKey": "", + "capabilities": { + "tools": true, + "uploadImage": true + } + } + ] + } + ``` + + + +Note: Not all models support function calling. Check the [OpenRouter models page](https://openrouter.ai/models) for specific model capabilities. diff --git a/docs/customize/model-providers/top-level/ollama.mdx b/docs/customize/model-providers/top-level/ollama.mdx index a592c289a0..6c2d4002f8 100644 --- a/docs/customize/model-providers/top-level/ollama.mdx +++ b/docs/customize/model-providers/top-level/ollama.mdx @@ -123,3 +123,42 @@ To configure a remote instance of Ollama, add the `"apiBase"` property to your m ``` + +## Model Capabilities + +Ollama models usually have their capabilities auto-detected correctly. However, if you're using custom model names or experiencing issues with tools/images not working, you can explicitly set capabilities: + + + + ```yaml title="config.yaml" + models: + - name: Custom Vision Model + provider: ollama + model: my-custom-llava + capabilities: + - tool_use # Enable if your model supports function calling + - image_input # Enable for vision models like llava + ``` + + + ```json title="config.json" + { + "models": [ + { + "title": "Custom Vision Model", + "provider": "ollama", + "model": "my-custom-llava", + "capabilities": { + "tools": true, + "uploadImage": true + } + } + ] + } + ``` + + + + +Most standard Ollama models (like llama3.1, mistral, etc.) support tool use by default. Vision models (like llava) also support image input. + diff --git a/docs/docs.json b/docs/docs.json index 04fc66d354..7ac29e2614 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -168,7 +168,8 @@ "customize/deep-dives/autocomplete", "customize/deep-dives/development-data", "customize/deep-dives/vscode-actions", - "customize/deep-dives/mcp" + "customize/deep-dives/mcp", + "customize/deep-dives/model-capabilities" ] }, { diff --git a/docs/features/agent/model-setup.mdx b/docs/features/agent/model-setup.mdx index a089c20c1a..88a22aa556 100644 --- a/docs/features/agent/model-setup.mdx +++ b/docs/features/agent/model-setup.mdx @@ -1,3 +1,37 @@ -The models you set up for Chat mode will be used with Agent mode if the model supports tool calling. +The models you set up for Chat mode will be used with Agent mode if the model supports tool calling. The recommended models and how to set them up can be found [here](/features/chat/model-setup). -The recommended models and how to set them up can be found [here](/features/chat/model-setup). +## System Message Tools + + + System message tools are experimental and may change in future releases. + + +Continue implements an innovative approach called **system message tools** that ensures consistent tool functionality across all models, regardless of their native capabilities. This allows Agent mode to work seamlessly with a wider range of models and providers. + +### How It Works + +Instead of relying solely on native tool calling APIs (which vary between providers), Continue converts tools into XML format and includes them in the system message. The model generates tool calls as structured XML within its response, which Continue then parses and executes. This approach provides: + +- **Universal compatibility** - Any model capable of following instructions can use tools, not just those with native tool support +- **Consistent behavior** - Tool calls work identically across OpenAI, Anthropic, local models, and others +- **Better reliability** - Models that struggle with native tools often perform better with system message tools +- **Seamless switching** - Change between providers without modifying your workflow + +### Recommended Models + +For the best Agent mode experience, we recommend models with strong reasoning and instruction-following capabilities: + +**Premium Models:** +- **Claude Sonnet 4** (Anthropic) - Our top recommendation for its exceptional tool use and reasoning +- **GPT-4** models (OpenAI) - Excellent native tool support +- **DeepSeek** models - Strong performance with competitive pricing + +**Local Models:** +While more limited in capabilities, these models can work with system message tools: +- Qwen2.5-Coder 32B - Best local option for Agent mode +- Devstral 27B - Good for code-specific tasks +- Smaller models (7B-13B) - May struggle with complex tool interactions + +### Configuration + +Agent mode automatically determines whether to use native or system message tools based on the model's capabilities. No additional configuration is required - simply select your model and Continue handles the rest. diff --git a/docs/features/autocomplete/model-setup.mdx b/docs/features/autocomplete/model-setup.mdx index 828aadf984..6f5b1d7999 100644 --- a/docs/features/autocomplete/model-setup.mdx +++ b/docs/features/autocomplete/model-setup.mdx @@ -43,3 +43,7 @@ This model provides good suggestions while keeping your code completely private. ## Need Help? If you're not seeing any completions or need more detailed configuration options, check out our comprehensive [autocomplete deep dive guide](/customize/deep-dives/autocomplete). + +## Model Compatibility + +To see a complete list of models and their capabilities, visit our [Model Capabilities guide](/customize/deep-dives/model-capabilities#model-compatibility-matrix). diff --git a/docs/features/chat/model-setup.mdx b/docs/features/chat/model-setup.mdx index 9e0a1c2132..06d61fc6dd 100644 --- a/docs/features/chat/model-setup.mdx +++ b/docs/features/chat/model-setup.mdx @@ -97,4 +97,8 @@ Their limited tool calling and reasoning capabilities will make it challenging t 2. Run the model with [Ollama](https://docs.continue.dev/guides/ollama-guide#using-ollama-with-continue-a-developers-guide) 3. Click `Reload config` in the assistant selector in the Continue IDE extension -For more detailed setup instructions, see [here](/getting-started/install#setup-your-first-model) +For more detailed setup instructions, see [here](/getting-started/install#setup-your-first-model). + +## Model Compatibility + +To see which models support specific features like tool use (for Agent mode) and image input, check out our [Model Capabilities guide](/customize/deep-dives/model-capabilities#model-compatibility-matrix). diff --git a/docs/features/edit/model-setup.mdx b/docs/features/edit/model-setup.mdx index 240e0613e1..8442015c95 100644 --- a/docs/features/edit/model-setup.mdx +++ b/docs/features/edit/model-setup.mdx @@ -10,3 +10,7 @@ We also recommend setting up an Apply model for the best Edit experience. - [Morph v3](https://hub.continue.dev/morphllm/morph-v2) - [Relace Instant Apply](https://hub.continue.dev/relace/instant-apply) + +## Model Compatibility + +For a complete overview of which models support various features, see our [Model Capabilities guide](/customize/deep-dives/model-capabilities#model-compatibility-matrix). diff --git a/docs/reference.mdx b/docs/reference.mdx index 836d8e7dce..879986df66 100644 --- a/docs/reference.mdx +++ b/docs/reference.mdx @@ -175,7 +175,11 @@ The `models` section defines the language models used in your configuration. Mod - `roles`: An array specifying the roles this model can fulfill, such as `chat`, `autocomplete`, `embed`, `rerank`, `edit`, `apply`, `summarize`. The default value is `[chat, edit, apply, summarize]`. Note that the `summarize` role is not currently used. -- `capabilities`: Array of strings denoting model capabilities, which will overwrite Continue's autodetection based on provider and model. Supported capabilities include `tool_use` and `image_input`. +- `capabilities`: Array of strings denoting model capabilities, which will overwrite Continue's autodetection based on provider and model. Supported capabilities include: + - `tool_use`: Enables function/tool calling support (required for Agent mode) + - `image_input`: Enables image upload and processing support + + Continue automatically detects these capabilities for most models, but you can override this when using custom deployments or if autodetection isn't working correctly. See the [Model Capabilities guide](/customize/deep-dives/model-capabilities) for detailed information. - `maxStopWords`: Maximum number of stop words allowed, to avoid API errors with extensive lists. diff --git a/docs/troubleshooting.mdx b/docs/troubleshooting.mdx index 60f1974428..d14f282d33 100644 --- a/docs/troubleshooting.mdx +++ b/docs/troubleshooting.mdx @@ -135,6 +135,64 @@ If you are having persistent errors with indexing, our recommendation is to rebu This can be accomplished using the following command: `Continue: Rebuild codebase index`. +### Agent mode is unavailable or tools aren't working + +If Agent mode is grayed out or tools aren't functioning properly, this is likely due to model capability configuration issues. + + + Continue uses system message tools as a fallback for models without native tool support, so most models should work with Agent mode automatically. + + +#### Check if your model has tool support + +1. Not all models support native tool/function calling, but Continue will automatically use system message tools as a fallback +2. Try adding `capabilities: ["tool_use"]` to your model config to force tool support +3. Verify your provider supports function calling or that system message tools are working correctly + +#### Tools Not Working + +If tools aren't being called: + +1. Ensure `tool_use` is in your capabilities +2. Check that your API endpoint actually supports function calling +3. Some providers may use different function calling formats + +#### Images Not Uploading + +If you can't upload images: + +1. Add `image_input` to capabilities +2. Ensure your model actually supports vision (e.g., gpt-4-vision, claude-3) +3. Check that your provider passes through image data + +#### Add capabilities + +If Continue's autodetection isn't working correctly, you can manually add capabilities in your `config.yaml`: + +```yaml +models: + - name: my-model + provider: openai + model: gpt-4 + capabilities: + - tool_use + - image_input +``` + +#### Verify with provider + +Some proxy services (like OpenRouter) or custom deployments may not preserve tool calling capabilities. Check your provider's documentation. + +#### Verifying Current Capabilities + +To see what capabilities Continue detected for your model: + +1. Check the mode selector tooltips - they indicate if tools are available +2. Try uploading an image - if disabled, the model lacks `image_input` +3. Check if Agent mode is available - requires `tool_use` + +See the [Model Capabilities guide](/customize/deep-dives/model-capabilities) for complete configuration details. + ### Android Studio - "Nothing to show" in Chat This can be fixed by selecting `Actions > Choose Boot runtime for the IDE` then selecting the latest version, and then restarting Android Studio. [See this thread](https://github.com/continuedev/continue/issues/596#issuecomment-1789327178) for details.