Add support for calling Anthropic models on Azure Foundry endpoints#58
Add support for calling Anthropic models on Azure Foundry endpoints#58JoseCSantos wants to merge 8 commits intoarcprize:mainfrom
Conversation
…opic_on_azure Add support for calling Anthropic models on Azure Foundry endpoints
|
Hi, in general, I'm a fan of getting Azure support on here. However, I would like to make it more explicit as to which Anthropic endpoint it goes to, rather than just relying on the environment variables to do this for us.
The model config should hold the information on how the model will be tested, not the env variables. Can you update this to make it more explicit to make it part of the model config? |
…for_anthropic_on_azure
…/github.com/JoseCSantos/arc-agi-benchmarking into jcsantos/add_support_for_anthropic_on_azure
|
Hello, Thanks for the feedback @gkamradt and sorry for the delay in sending an update to this PR. In the coming days/weeks I may send follow on PRs for other model vendors in Azure, namely OpenAI. Have tested Opus 4.5 with 64k reasoning budget and got: Final Score: 34.58% (41.50/120) Looking at the official ARC-AGI-2 leaderboard we have for Opus-4.5-64k: While the cost is very similar ($2.40 vs $2.48), the final score is 3.0% lower, though the reported 37.6% is still within the 95th confidence interval [26.4%, 42.8%] of my run, so benchmark seems correct. Thanks |
This pull request enhances the
AnthropicAdapterto support both direct Anthropic API access and Azure-hosted Anthropic endpoints, improving flexibility in how API credentials and endpoints are configured. It also introduces logic to adjust the model name for Azure deployments and ensures consistent usage of the correct model name throughout the adapter.Provider configuration and model selection:
ANTHROPIC_API_KEY) or Azure-hosted credentials (AZURE_ANTHROPIC_API_KEYandAZURE_ANTHROPIC_ENDPOINT). The adapter now chooses the appropriate connection mode based on available environment variables, with error handling for missing configuration._get_model_namemethod to strip date suffixes from the model name when using Azure endpoints, ensuring compatibility with Azure deployment naming conventions.Consistent model usage:
chat_completionandchat_completion_stream) to use the new_get_model_namemethod, guaranteeing that the correct model identifier is used depending on the deployment type. [1] [2]Internal improvements:
import reto support model name manipulation.