fix(token-counting): resolve os.environ/ prefix and include system/tools in Anthropic API #17486
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ols in Anthropic API This PR fixes three bugs in count_tokens_with_anthropic_api that cause inaccurate token counting for Anthropic models: 1. **os.environ/ prefix not resolved**: When deployment config uses `api_key: os.environ/ANTHROPIC_API_KEY`, the literal string was passed to the Anthropic SDK, causing auth failure and silent fallback to tiktoken (which undercounts by ~2-15%). 2. **system prompt ignored**: The system prompt was not passed to the Anthropic count_tokens API, resulting in undercount. 3. **tools ignored**: Tool definitions were not passed to the Anthropic count_tokens API, resulting in undercount. ## Changes - `litellm/proxy/utils.py`: Added os.environ/ resolution via get_secret_str, added system and tools parameters to count_tokens_with_anthropic_api - `litellm/proxy/_types.py`: Added system and tools fields to TokenCountRequest - `litellm/proxy/anthropic_endpoints/endpoints.py`: Extract system/tools from request - `litellm/proxy/proxy_server.py`: Pass system/tools through to provider counter - `litellm/llms/anthropic/common_utils.py`: Accept system/tools in AnthropicTokenCounter - `litellm/llms/base_llm/base_utils.py`: Updated BaseTokenCounter interface ## Test Results With a real 195-message conversation: - Messages only: FIXED matches Direct API EXACTLY (185,049 tokens) - Messages + System: FIXED matches Direct API EXACTLY (185,065 tokens) - Messages + System + Tools: FIXED matches Direct API EXACTLY (185,719 tokens) - BROKEN undercounts by 5,045 tokens (2.7%) due to tiktoken fallback
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relevant issues
Fixes token counting inaccuracies for deployments using
os.environ/prefix syntax in API key configuration.Pre-Submission checklist
tests/directoryType
Bug Fix
Changes
This PR fixes three bugs in
count_tokens_with_anthropic_apithat cause inaccurate token counting for Anthropic models:os.environ/ prefix not resolved: When deployment config uses
api_key: os.environ/ANTHROPIC_API_KEY, the literal string was passed to the Anthropic SDK, causing authentication failure and silent fallback to tiktoken.system prompt ignored: The system prompt was not passed to the Anthropic count_tokens API.
tools ignored: Tool definitions were not passed to the Anthropic count_tokens API.
Files changed
litellm/proxy/utils.py: Added os.environ/ resolution via get_secret_str, added system and tools parameterslitellm/proxy/_types.py: Added system and tools fields to TokenCountRequestlitellm/proxy/anthropic_endpoints/endpoints.py: Extract system/tools from requestlitellm/proxy/proxy_server.py: Pass system/tools through to provider counterlitellm/llms/anthropic/common_utils.py: Accept system/tools in AnthropicTokenCounterlitellm/llms/base_llm/base_utils.py: Updated BaseTokenCounter interfaceTests
Added 3 new tests in
tests/proxy_unit_tests/test_proxy_token_counter.py:test_count_tokens_with_anthropic_api_resolves_os_environ_prefixtest_count_tokens_with_anthropic_api_passes_system_and_toolstest_anthropic_endpoint_extracts_system_and_toolsVerified with real conversation (195 messages):
Fixed version matches direct Anthropic API exactly. Broken version undercounts by 5,045 tokens (2.7%).