fix(tutorial): lowercase both operands in the Exercise 3.1 grader - #174
Open
shoemoney wants to merge 1 commit into
Open
fix(tutorial): lowercase both operands in the Exercise 3.1 grader#174shoemoney wants to merge 1 commit into
shoemoney wants to merge 1 commit into
Conversation
`grade_exercise` in Chapter 3 tests `"incorrect" in text` without lowercasing,
while the very next operand in the same expression uses `text.lower()`. The
exercise asks the learner to make Claude judge `2x - 3 = 9 / 2x = 6 / x = 3`
as incorrectly solved, and a response that opens with a capitalised "Incorrect"
— the most natural way to start that sentence — is graded as a failure.
grade_exercise("Incorrect. 2x - 3 = 9 means 2x = 12, so x = 6.") -> False
grade_exercise("No, the solution is INCORRECT.") -> False
Every other substring grader in the tutorial already lowercases the haystack
(`"hola" in text.lower()`, `re.search("brown", text.lower())`); the ones that
do not are matching deliberately capitalised strings like "Warrior".
Applied to all three copies of the chapter (Anthropic 1P, AmazonBedrock/anthropic,
AmazonBedrock/boto3), which carry the same line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Exercise 3.1 grader in Chapter 3 lowercases one operand and not the other, so a correct answer is graded as a failure.
The exercise asks the learner to get Claude to judge this as incorrectly solved:
A response that opens with a capitalised "Incorrect" — the most natural way to begin that sentence, and what a role-prompted math teacher tends to produce — misses the first operand entirely, and "Incorrect." does not contain "not correct" either:
Three of those four are correct answers marked wrong. A learner who solves the exercise as intended sees
This exercise has been correctly solved: Falseand reasonably concludes their prompt is the problem.The change
One character per file:
text→text.lower()on the first operand.This matches what the second operand in the same expression already does, and the convention everywhere else in the tutorial —
"hola" in text.lower()in Chapter 2,re.search("brown", text.lower())in Chapter 4. The graders that don't lowercase are matching deliberately capitalised strings ("Warrior","49-fold"), which is not the case here.Applied to all three copies of the chapter, which carry the identical line:
prompt_engineering_interactive_tutorial/Anthropic 1P/03_Assigning_Roles_Role_Prompting.ipynbprompt_engineering_interactive_tutorial/AmazonBedrock/anthropic/03_Assigning_Roles_Role_Prompting.ipynbprompt_engineering_interactive_tutorial/AmazonBedrock/boto3/03_Assigning_Roles_Role_Prompting.ipynbTesting
Ran the grader verbatim before and after. After the change all five phrasings of a correct answer pass, and responses that judge the equation as correctly solved still return
False— "solved correctly" contains neitherincorrectnornot correct, so no false positives are introduced.Only the one line changed in each notebook:
git diff --numstatis1 1per file, with no cell-output or metadata churn.Not in this PR
real_world_prompting/02_medical_prompt.ipynbhas an unrelated defect —extract_action_itemspassesNoneintojson.loadswhen the<summary>tags are missing, and the resultingTypeErroris not caught by the surroundingexcept json.JSONDecodeError, so it aborts the batch loop. Separate concern, happy to send it separately if useful.