Skip to content

fix(tutorial): lowercase both operands in the Exercise 3.1 grader - #174

Open
shoemoney wants to merge 1 commit into
anthropics:masterfrom
shoemoney:fix/role-prompting-grader-case
Open

fix(tutorial): lowercase both operands in the Exercise 3.1 grader#174
shoemoney wants to merge 1 commit into
anthropics:masterfrom
shoemoney:fix/role-prompting-grader-case

Conversation

@shoemoney

Copy link
Copy Markdown

Summary

The Exercise 3.1 grader in Chapter 3 lowercases one operand and not the other, so a correct answer is graded as a failure.

def grade_exercise(text):
    if "incorrect" in text or "not correct" in text.lower():

The exercise asks the learner to get Claude to judge this as incorrectly solved:

2x - 3 = 9
2x = 6
x = 3

A response that opens with a capitalised "Incorrect" — the most natural way to begin that sentence, and what a role-prompted math teacher tends to produce — misses the first operand entirely, and "Incorrect." does not contain "not correct" either:

grade_exercise("Incorrect. 2x - 3 = 9 means 2x = 12, so x = 6.")  # -> False
grade_exercise("No, the solution is INCORRECT.")                  # -> False
grade_exercise("Incorrect!")                                      # -> False
grade_exercise("incorrect - the second step is wrong.")           # -> True

Three of those four are correct answers marked wrong. A learner who solves the exercise as intended sees This exercise has been correctly solved: False and reasonably concludes their prompt is the problem.

The change

One character per file: texttext.lower() on the first operand.

-    if "incorrect" in text or "not correct" in text.lower():
+    if "incorrect" in text.lower() or "not correct" in text.lower():

This matches what the second operand in the same expression already does, and the convention everywhere else in the tutorial — "hola" in text.lower() in Chapter 2, re.search("brown", text.lower()) in Chapter 4. The graders that don't lowercase are matching deliberately capitalised strings ("Warrior", "49-fold"), which is not the case here.

Applied to all three copies of the chapter, which carry the identical line:

  • prompt_engineering_interactive_tutorial/Anthropic 1P/03_Assigning_Roles_Role_Prompting.ipynb
  • prompt_engineering_interactive_tutorial/AmazonBedrock/anthropic/03_Assigning_Roles_Role_Prompting.ipynb
  • prompt_engineering_interactive_tutorial/AmazonBedrock/boto3/03_Assigning_Roles_Role_Prompting.ipynb

Testing

Ran the grader verbatim before and after. After the change all five phrasings of a correct answer pass, and responses that judge the equation as correctly solved still return False — "solved correctly" contains neither incorrect nor not correct, so no false positives are introduced.

Only the one line changed in each notebook: git diff --numstat is 1 1 per file, with no cell-output or metadata churn.

Not in this PR

real_world_prompting/02_medical_prompt.ipynb has an unrelated defect — extract_action_items passes None into json.loads when the <summary> tags are missing, and the resulting TypeError is not caught by the surrounding except json.JSONDecodeError, so it aborts the batch loop. Separate concern, happy to send it separately if useful.

`grade_exercise` in Chapter 3 tests `"incorrect" in text` without lowercasing,
while the very next operand in the same expression uses `text.lower()`. The
exercise asks the learner to make Claude judge `2x - 3 = 9 / 2x = 6 / x = 3`
as incorrectly solved, and a response that opens with a capitalised "Incorrect"
— the most natural way to start that sentence — is graded as a failure.

    grade_exercise("Incorrect. 2x - 3 = 9 means 2x = 12, so x = 6.")  -> False
    grade_exercise("No, the solution is INCORRECT.")                  -> False

Every other substring grader in the tutorial already lowercases the haystack
(`"hola" in text.lower()`, `re.search("brown", text.lower())`); the ones that
do not are matching deliberately capitalised strings like "Warrior".

Applied to all three copies of the chapter (Anthropic 1P, AmazonBedrock/anthropic,
AmazonBedrock/boto3), which carry the same line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant