Commit0: bug 526 impact on old results #545
Description
This issue tracks the impact of #526 on old evaluation results for Commit0.
We analyzed the commit0 results from https://github.com/OpenHands/openhands-index-results/tree/main/results for cachetools and parsel to see if they had errors that marked them as unresolved due to the test command issues.
Here are the results:
| Model Name | cachetools error | parsel error |
|---|---|---|
| DeepSeek-V3.2-Reasoner | No | Yes (exit code 127) |
| GLM-4.7 | Yes (exit code 1, likely ImportError) | No |
| GLM-5 | Yes (exit code 1, likely ImportError) | No |
| GPT-5.2 | No | Yes (exit code 127) |
| GPT-5.2-Codex | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| GPT-5.4 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Gemini-3-Flash | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Gemini-3-Pro | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Gemini-3.1-Pro | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Kimi-K2-Thinking | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Kimi-K2.5 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| MiniMax-M2.1 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| MiniMax-M2.5 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Nemotron-3-Nano | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Qwen3-Coder-480B | No | No |
| Qwen3-Coder-Next | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| Qwen3.5-Flash | Yes (exit code 1, likely ImportError) | No |
| claude-opus-4-5 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| claude-opus-4-6 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| claude-sonnet-4-5 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |
| claude-sonnet-4-6 | Yes (exit code 1, likely ImportError) | Yes (exit code 127) |