Skip to content

docs: add model capabilities guide and update provider documentation with tool_use #6901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Aug 5, 2025

Conversation

bdougie
Copy link
Collaborator

@bdougie bdougie commented Jul 30, 2025

Description

  • Add new model-capabilities.mdx deep dive guide explaining how to configure tool and image support
  • Update OpenRouter provider documentation with capability configuration examples
  • Update Ollama provider documentation with tool_use support details
  • Enhance reference documentation with clearer capability guidance
  • Improve troubleshooting documentation for capability-related issues

[ What changed? Feel free to be brief. ]

Checklist

  • I've read the contributing guide
  • The relevant docs, if any, have been updated or created
  • The relevant tests, if any, have been updated or created

Screen recording or screenshot

[ When applicable, please include a short screen recording or screenshot - this makes it much easier for us as contributors to review and understand your changes. See this PR as a good example. ]

There is also a compatibility matrix added. I did generate this, but expect to get feedback on actual compatibilities in review as this is not my expertise.

Screenshot 2025-08-05 at 10 55 45 AM

Tests

[ What tests were added or updated to ensure the changes work as expected? ]

  1. View the new compatibility matrix - https://continue-docs-bdougie-con-2740.mintlify.app/customize/deep-dives/model-capabilities.

Summary by cubic

Added a new guide on model capabilities and updated provider documentation to help users configure tool and image support for custom and proxy models.

  • Docs Updates
    • Added a deep dive guide explaining model capabilities and manual overrides.
    • Updated OpenRouter and Ollama docs with capability configuration examples.
    • Improved reference and troubleshooting docs with clearer guidance on capability issues.

- Add new model-capabilities.mdx deep dive guide
- Update OpenRouter provider documentation
- Update Ollama provider documentation
- Update reference documentation
- Update troubleshooting documentation
- Add model-capabilities.mdx to deep-dives section in docs.json
- Add comprehensive compatibility matrix for popular models
- Include OpenAI, Anthropic, Google, Mistral, DeepSeek, and Ollama models
- Document feature support for Chat, Apply, Edit, Tool Use, and Image Input
- Add GitHub edit link for community contributions
@bdougie bdougie force-pushed the bdougie/con-2740 branch from 6008d3c to 2de4a4c Compare July 30, 2025 23:43
@bdougie bdougie changed the title docs: add model capabilities guide and update provider documentation docs: add model capabilities guide and update provider documentation with tool_use Jul 31, 2025
- Remove migration section reference to non-existent page
- Align table headers with left alignment for better readability
- Clean up document structure by removing broken cross-reference
- Enhanced model capabilities documentation with more detailed explanations
- Added model setup information across agent, autocomplete, chat, and edit features
- Improved documentation structure and clarity for better user understanding
@bdougie bdougie force-pushed the bdougie/con-2740 branch from 5f7b99e to a23cd28 Compare July 31, 2025 19:22
@bdougie bdougie marked this pull request as ready for review July 31, 2025 21:31
@bdougie bdougie requested a review from a team as a code owner July 31, 2025 21:31
@bdougie bdougie removed the request for review from a team July 31, 2025 21:31
Copy link
Contributor

@BekahHW BekahHW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@bdougie bdougie requested a review from RomneyDa August 4, 2025 22:57
You can explicitly set the capabilities you want the model to use. This disables autodetection for any other capabilities.

<Note>
You cannot configure a model to use no capabilities.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to have no capabilities? Maybe say you cannot override autodetection


Continue automatically detects capabilities based on your provider and model name. For example:

- **OpenAI**: GPT-4 models have tool support, GPT-3.5 doesn't
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting, gpt-3.5 turbo models actually support function calling, we may want to change this support but at least this example might confuse users who know that

<Info>
The `tool_use` capability is for native tool/function calling support. The
model must actually support tools for this to work. If your model doesn't
support native tools, Continue will automatically use system message tools
Copy link
Collaborator

@RomneyDa RomneyDa Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The wording on this is a bit confusing. System message tools are currently experimental and will not be automatically used as a fallback. capabilities only applies to native tools, it does not impact system message tool use whether it is set or not. An exception is openrouter, which we did a hot fix for

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot 2025-08-05 at 9 26 12 AM

added a warn callout

| GPT-4o | ✅ | ✅ |
| GPT-4 Turbo | ✅ | ✅ |
| GPT-4 | ✅ | ❌ |
| GPT-3.5 Turbo | ❌ | ❌ |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above


### Notes

- **Tool Use**: Function calling for Agent mode (required for Agent mode)
Copy link
Collaborator

@RomneyDa RomneyDa Aug 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tool_use property is not required for agent mode (tools are), wording is a bit confusing

### Notes

- **Tool Use**: Function calling for Agent mode (required for Agent mode)
- **Image Input**: Processing screenshots and images
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "screenshots and images" is confusing because a screenshot is an image and feels oddly specific here

### Recommended Models

For the best Agent mode experience, we recommend models with strong reasoning and instruction-following capabilities:

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might recommend against using these models with system message tools because they are trained to use native tools and very effective with them. Besides deepseek

2. Ensure your model actually supports vision (e.g., gpt-4-vision, claude-3)
3. Check that your provider passes through image data

#### Override capabilities
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe *add capabilities, since there's no override functionality

@bdougie
Copy link
Collaborator Author

bdougie commented Aug 5, 2025

Hey @BekahHW and @RomneyDa, the requested changes have been implemented in the latest commits:

  • Clarified model capabilities configuration and detection
  • Updated the compatibility matrix columns as discussed

Please let me know if there's anything else needed!


Generated with cn

Copy link
Collaborator

@RomneyDa RomneyDa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@@ -175,7 +175,11 @@ The `models` section defines the language models used in your configuration. Mod

- `roles`: An array specifying the roles this model can fulfill, such as `chat`, `autocomplete`, `embed`, `rerank`, `edit`, `apply`, `summarize`. The default value is `[chat, edit, apply, summarize]`. Note that the `summarize` role is not currently used.

- `capabilities`: Array of strings denoting model capabilities, which will overwrite Continue's autodetection based on provider and model. Supported capabilities include `tool_use` and `image_input`.
- `capabilities`: Array of strings denoting model capabilities, which will overwrite Continue's autodetection based on provider and model. Supported capabilities include:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tool_use does not overwrite autodetection, as noted above

@github-project-automation github-project-automation bot moved this from Todo to In Progress in Issues and PRs Aug 5, 2025
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 5, 2025
@bdougie bdougie merged commit 33fa558 into main Aug 5, 2025
38 checks passed
@bdougie bdougie deleted the bdougie/con-2740 branch August 5, 2025 18:16
@github-project-automation github-project-automation bot moved this from In Progress to Done in Issues and PRs Aug 5, 2025
@github-actions github-actions bot locked and limited conversation to collaborators Aug 5, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm This PR has been approved by a maintainer size:L This PR changes 100-499 lines, ignoring generated files.
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

3 participants