Use the last reasoning delimiter

[RerankerGuo]

What does this PR do?

reasoning_accuracy_reward documents the final answer as the text after the last occurrence of any configured reasoning delimiter. The current loop instead stops at the first delimiter type from reasoning_delimiters that appears anywhere, even if another configured delimiter occurs later in the completion.

This change computes the end positions of all matching delimiters and slices after the textually last one. It preserves the existing behavior for repeated occurrences of one delimiter and for completions with no matching delimiter.

Fixes #6661

PR #6222 also touches accuracy_rewards.py, but it adds separate reward factories and does not modify this delimiter-selection block.

Verification

The regression test replaces only the optional math parser/verifier so it runs even when math_verify is not installed. It demonstrates that argument-list order no longer overrides completion order.

Before the fix:

1 failed in 12.07s

After the fix:

1 passed in 4.20s
19 passed, 20 skipped in 4.86s

Commands:

pytest -q tests/test_rewards.py::TestReasoningAccuracyReward::test_uses_last_matching_reasoning_delimiter
pytest -q tests/test_rewards.py
pre-commit run --files trl/rewards/accuracy_rewards.py tests/test_rewards.py
git diff --check

The 20 skipped tests are existing tests gated on optional math dependencies. The new regression test executed without that extra. Pre-commit passed Ruff check, Ruff format, and doc-builder style.

Before submitting

• This PR fixes a typo or improves the docs (not applicable).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? See Reasoning reward uses delimiter-list order instead of text order #6661.
• Did you make sure to update the documentation with your changes? Existing documentation already states the corrected behavior.
• Did you write any new necessary tests?

AI writing disclosure

We welcome the use of AI tools to help with contributions. For transparency and to help us improve our review process, please indicate the level of AI involvement in this PR.

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone familiar with the reward utilities can review this focused change.

---

Note

Low Risk
Small, localized change to reward parsing logic with a targeted test; behavior only shifts for multi-delimiter completions where list order previously disagreed with text order.

Overview
reasoning_accuracy_reward now takes the final answer from the text after the latest matching delimiter in the completion, not after the first delimiter type listed in reasoning_delimiters. Delimiter selection uses rfind across all configured delimiters and slices at the maximum end position; missing delimiters still yield a 0.0 incomplete-reasoning penalty.

A regression test mocks math_verify so it runs without optional deps and checks that when </think> appears before </analysis>, grading uses the text after </analysis>.

^{Reviewed by Cursor Bugbot for commit 1771628. Bugbot is set up for automated code reviews on this repo. Configure here.}