Referee model selection

[ketchup8866]

AS for a safety check, which external model should I choose as the referee model? Any recommendations?

[zonashi]

1、Recommended Choices for MCP Scan

• GLM4.5
• DeepSeek-V3.1
• Kimi-K2-Instruct
• Qwen3-Coder-480B
• Hunyuan-Turbos

2、Recommended Choices for Jailbreak Evaluation Models

When working with a custom dataset, selecting an appropriate safety evaluation model can significantly improve the accuracy of automated assessments. You can balance model selection from two dimensions: language and scenario.

Language

• Chinese Recommendation:

  • qwen3-max (best performance)
  • qwen3-235b-a22b-2507 (cost-effective choice)

• English Recommendation:

  • claude-opus-4.1 (best performance)
  • claude-sonnet-4 (very good performance)
  • gemini-2.0-flash (cost-effective choice)

Scenario

• Politically sensitive content testing:
  Do not choose Gemini models. Instead, prioritize domestic models such as hunyuan-turbos or qwen3. Cloud-based API calls yield better results.

• National, regional, or racial bias testing:
  Gemini models perform best.

• Dangerous weapons or high-risk behavior testing:
  Claude models perform best. For cost-effectiveness, Gemini models are also an option.