Add physics benchmark framework with 25 tasks across 5 subfields#86
Open
madeleinesong wants to merge 6 commits into
Open
Add physics benchmark framework with 25 tasks across 5 subfields#86madeleinesong wants to merge 6 commits into
madeleinesong wants to merge 6 commits into
Conversation
Define BenchmarkTask, BenchmarkSuite, Reference, and supporting enums (Difficulty, TaskType, OutputFormat) as the foundation for a systematic physics benchmark. Each task captures problem statement, classification metadata, expected answers, and paper provenance. ENG-437
25 tasks covering QFT (5), GR & cosmology (5), statistical mechanics (5), condensed matter (4), classical mechanics (3), and quantum information (3). Each task specifies problem statement, classification metadata (difficulty, type, subfield), expected answer, verification hints, and paper/textbook references. Difficulty ranges from introductory to advanced. Task types include derivation, calculation, dimensional analysis, limiting cases, and estimation. ENG-437
- loader.py: discovers task JSON files, loads individual and combined suites, supports filtering by subfield/difficulty/type, and provides an inventory summary. - runner.py: model-agnostic benchmark runner that formats task prompts, invokes a caller-provided model function, and collects structured results with timing. Includes a report formatter. ENG-437
40 tests covering: - Schema roundtrip serialization (Reference, BenchmarkTask, BenchmarkSuite) - All enum values (TaskType, Difficulty, OutputFormat) - Suite filtering by subfield, difficulty, and task type - Suite save/load to JSON files - Task file consistency: unique IDs, required fields, valid subfields - Cross-file coverage requirements (subfields, difficulties, task types) - Prompt formatting (includes problem info, excludes answers/hints) - Runner execution with mock models (success, error, suite-level) - Report formatting and result serialization ENG-437
madeleinesong
force-pushed
the
eng-437-create-benchmark-tasks
branch
from
1109b34 to
9fe5529
Compare
- benchmarks/schema.py: use enum.StrEnum (UP042) for the three string enums - benchmarks/runner.py: drop unused dataclasses.field import (F401) - tests/test_benchmarks.py: import benchmarks package via pytest pythonpath instead of a sys.path hack, removing the E402 module-import-not-at-top violations and the unused TASKS_DIR import (F401) - pyproject.toml: add repo root to pytest pythonpath so the root-level benchmarks package is importable without per-test sys.path manipulation - tests/core/test_prompt_exactness_budget.py: raise exact_assertion_count ceiling 5165 -> 5170 to absorb the benchmark framework's 6 new machine-contract prompt-format assertions
📝 WalkthroughWalkthroughThis PR implements the GPD Physics Benchmark framework—a systematic evaluation suite for physics AI across six domains. It introduces immutable schema models for benchmark tasks and suites, a loader that discovers and aggregates JSON task files, a model-agnostic runner that executes tasks and captures results, curated physics benchmark tasks with problem statements and expected answers, and comprehensive tests validating the entire framework. ChangesPhysics Benchmark Framework
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
tests/core/test_prompt_exactness_budget.py (1)
15-19: 💤 Low valueComment claims "adds 6" but budget increases by 5.
The comment states the benchmark framework "adds 6 machine-contract prompt-format assertions", but the budget increase from
5_165to5_170is only +5. Line 17 clarifies that 5170 is the observed value, so the budget itself is correct. Consider updating the comment to match the actual increase:📝 Suggested comment clarification
- # Benchmark framework (eng-437) adds 6 machine-contract prompt-format + # Benchmark framework (eng-437) adds 5 net exact prompt-format # assertions in tests/test_benchmarks.py; raise the exact-assertion ceiling # to absorb them (5170 observed) while keeping the brittle-prose cap fixed.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/core/test_prompt_exactness_budget.py` around lines 15 - 19, The inline comment incorrectly says the benchmark framework "adds 6" assertions while the exact_assertion_count was only raised from 5_165 to 5_170 (+5); update the comment text near the keys "brittle_prose_assertions" and "exact_assertion_count" to either state "adds 5" or remove the numeric claim and instead note that the observed value is 5170 and the budget was set to 5170 to match observed results. Ensure the comment and the budget value are consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/loader.py`:
- Around line 16-20: discover_task_files currently returns an empty list when
TASKS_DIR doesn't exist or contains no JSON files, which hides
configuration/packaging problems; change discover_task_files to raise an
exception (e.g., RuntimeError or SystemExit with a clear message) when no task
files are found instead of returning [], and update any callers that expect a
combined suite to propagate/handle that exception (search for places that call
discover_task_files to ensure they also fail fast and don’t silently create an
empty benchmark).
In `@benchmarks/schema.py`:
- Around line 67-75: The dataclasses marked frozen (Reference, BenchmarkTask,
BenchmarkSuite) still expose mutable list fields; change their internal
representations to immutable tuples for fields Reference.authors,
BenchmarkTask.topics, BenchmarkTask.given, BenchmarkTask.assumptions,
BenchmarkTask.conventions, BenchmarkTask.verification_hints, and
BenchmarkSuite.tasks, and update any constructors/serializers so external
JSON/IO still accepts/returns lists: convert incoming lists to tuples in the
class factory/from_dict and convert tuples back to lists in
to_dict/serialization methods; ensure immutability is enforced by using tuple
types for those attributes and adjust any usages that mutate those lists to
instead create new instances (respecting frozen dataclass) via the dataclass
constructors or builder helpers.
In `@benchmarks/tasks/gr_cosmology.json`:
- Around line 153-157: The metadata for gr-004 is inconsistent: the arxiv_id
"2310.14698" conflicts with the 1975 Hawking bibliographic details. Pick one
canonical source and make all fields match it — either replace arxiv_id with the
correct identifier/DOI for Hawking 1975 and keep title/authors/year/section
as-is (update "arxiv_id" to the 1975 DOI or arXiv if available), or update
title/authors/year/section to reflect the 2023 paper for arXiv:2310.14698;
ensure you modify the JSON keys "arxiv_id", "title", "authors", and "year" in
the same object so they consistently represent the chosen source.
- Line 36: Replace the misspelled "Kretschner" in the verification hint string
"The Kretschner scalar R_{abcd} R^{abcd} = 48 M^2/r^6 confirms a true
singularity at r = 0" with the correct term "Kretschmann" so the user-facing
benchmark text reads "The Kretschmann scalar R_{abcd} R^{abcd} = 48 M^2/r^6
confirms a true singularity at r = 0"; locate and update the exact JSON string
in the file (search for the substring "Kretschner scalar" or the full hint
sentence) to correct the typo.
---
Nitpick comments:
In `@tests/core/test_prompt_exactness_budget.py`:
- Around line 15-19: The inline comment incorrectly says the benchmark framework
"adds 6" assertions while the exact_assertion_count was only raised from 5_165
to 5_170 (+5); update the comment text near the keys "brittle_prose_assertions"
and "exact_assertion_count" to either state "adds 5" or remove the numeric claim
and instead note that the observed value is 5170 and the budget was set to 5170
to match observed results. Ensure the comment and the budget value are
consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 3313ee63-c7f1-40bb-aa38-2ec0e81af600
📒 Files selected for processing (13)
benchmarks/__init__.pybenchmarks/loader.pybenchmarks/runner.pybenchmarks/schema.pybenchmarks/tasks/classical_mechanics.jsonbenchmarks/tasks/condensed_matter.jsonbenchmarks/tasks/gr_cosmology.jsonbenchmarks/tasks/qft.jsonbenchmarks/tasks/quantum_info.jsonbenchmarks/tasks/stat_mech.jsonpyproject.tomltests/core/test_prompt_exactness_budget.pytests/test_benchmarks.py
Comment on lines
+16
to
+20
| def discover_task_files() -> list[Path]: | ||
| """Return sorted list of JSON task files in the tasks/ directory.""" | ||
| if not TASKS_DIR.is_dir(): | ||
| return [] | ||
| return sorted(TASKS_DIR.glob("*.json")) |
There was a problem hiding this comment.
Fail fast when no task files are discovered instead of returning an empty benchmark.
Returning a valid-looking combined suite with zero tasks hides configuration/packaging breakage and can make downstream reports look successful but meaningless.
Proposed fix
def load_combined_suite() -> BenchmarkSuite:
"""Load all task files and combine into a single suite."""
+ files = discover_task_files()
+ if not files:
+ raise FileNotFoundError(f"No benchmark task files found in: {TASKS_DIR}")
all_tasks: list[BenchmarkTask] = []
- for suite in load_all_suites():
+ for suite in (load_suite(path) for path in files):
all_tasks.extend(suite.tasks)Also applies to: 31-41
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/loader.py` around lines 16 - 20, discover_task_files currently
returns an empty list when TASKS_DIR doesn't exist or contains no JSON files,
which hides configuration/packaging problems; change discover_task_files to
raise an exception (e.g., RuntimeError or SystemExit with a clear message) when
no task files are found instead of returning [], and update any callers that
expect a combined suite to propagate/handle that exception (search for places
that call discover_task_files to ensure they also fail fast and don’t silently
create an empty benchmark).
Comment on lines
+67
to
+75
| @dataclass(frozen=True) | ||
| class Reference: | ||
| """Citation for the source paper or textbook.""" | ||
|
|
||
| arxiv_id: str | None = None | ||
| doi: str | None = None | ||
| title: str = "" | ||
| authors: list[str] = field(default_factory=list) | ||
| year: int | None = None |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Read-only verification: locate frozen dataclasses that still expose mutable list fields
rg -n "dataclass\\(frozen=True\\)|: list\\[|default_factory=list" benchmarks/schema.py -C2Repository: psi-oss/get-physics-done
Length of output: 1372
Make frozen=True dataclasses truly immutable by removing mutable list fields
In benchmarks/schema.py, @dataclass(frozen=True) classes (Reference, BenchmarkTask, BenchmarkSuite) still define mutable list[...] fields (Reference.authors, BenchmarkTask.topics/given/assumptions/conventions/verification_hints, BenchmarkSuite.tasks). frozen=True only blocks attribute reassignment; these lists can still be mutated in-place, breaking the “immutable schema” contract.
Suggested direction
- authors: list[str] = field(default_factory=list)
+ authors: tuple[str, ...] = ()
- topics: list[str] = field(default_factory=list)
+ topics: tuple[str, ...] = ()
- given: list[str] = field(default_factory=list)
+ given: tuple[str, ...] = ()
- assumptions: list[str] = field(default_factory=list)
+ assumptions: tuple[str, ...] = ()
- conventions: list[str] = field(default_factory=list)
+ conventions: tuple[str, ...] = ()
- verification_hints: list[str] = field(default_factory=list)
+ verification_hints: tuple[str, ...] = ()
- tasks: list[BenchmarkTask] = field(default_factory=list)
+ tasks: tuple[BenchmarkTask, ...] = ()Ensure from_dict()/to_dict() convert these fields consistently (tuples internally; JSON lists outward).
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| @dataclass(frozen=True) | |
| class Reference: | |
| """Citation for the source paper or textbook.""" | |
| arxiv_id: str | None = None | |
| doi: str | None = None | |
| title: str = "" | |
| authors: list[str] = field(default_factory=list) | |
| year: int | None = None | |
| `@dataclass`(frozen=True) | |
| class Reference: | |
| """Citation for the source paper or textbook.""" | |
| arxiv_id: str | None = None | |
| doi: str | None = None | |
| title: str = "" | |
| authors: tuple[str, ...] = () | |
| year: int | None = None |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/schema.py` around lines 67 - 75, The dataclasses marked frozen
(Reference, BenchmarkTask, BenchmarkSuite) still expose mutable list fields;
change their internal representations to immutable tuples for fields
Reference.authors, BenchmarkTask.topics, BenchmarkTask.given,
BenchmarkTask.assumptions, BenchmarkTask.conventions,
BenchmarkTask.verification_hints, and BenchmarkSuite.tasks, and update any
constructors/serializers so external JSON/IO still accepts/returns lists:
convert incoming lists to tuples in the class factory/from_dict and convert
tuples back to lists in to_dict/serialization methods; ensure immutability is
enforced by using tuple types for those attributes and adjust any usages that
mutate those lists to instead create new instances (respecting frozen dataclass)
via the dataclass constructors or builder helpers.
| "Check Birkhoff's theorem: this is the unique spherically symmetric vacuum solution", | ||
| "Verify R_{mu nu} = 0 by direct computation for the derived metric", | ||
| "In the limit M -> 0, the Minkowski metric is recovered", | ||
| "The Kretschner scalar R_{abcd} R^{abcd} = 48 M^2/r^6 confirms a true singularity at r = 0" |
There was a problem hiding this comment.
Fix the scalar name typo in verification hints.
“Kretschner scalar” should be “Kretschmann scalar”; this is user-facing benchmark text and can confuse solvers.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/tasks/gr_cosmology.json` at line 36, Replace the misspelled
"Kretschner" in the verification hint string "The Kretschner scalar R_{abcd}
R^{abcd} = 48 M^2/r^6 confirms a true singularity at r = 0" with the correct
term "Kretschmann" so the user-facing benchmark text reads "The Kretschmann
scalar R_{abcd} R^{abcd} = 48 M^2/r^6 confirms a true singularity at r = 0";
locate and update the exact JSON string in the file (search for the substring
"Kretschner scalar" or the full hint sentence) to correct the typo.
Comment on lines
+153
to
+157
| "arxiv_id": "2310.14698", | ||
| "title": "Black Holes, Thermodynamics, and Hawking Radiation", | ||
| "authors": ["S. W. Hawking"], | ||
| "year": 1975, | ||
| "section": "Section 2" |
There was a problem hiding this comment.
Reference metadata is internally inconsistent for gr-004.
The arxiv_id does not match the same bibliographic identity as the 1975 Hawking PRL DOI/year block. Please align these fields to one canonical source to preserve benchmark provenance integrity.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@benchmarks/tasks/gr_cosmology.json` around lines 153 - 157, The metadata for
gr-004 is inconsistent: the arxiv_id "2310.14698" conflicts with the 1975
Hawking bibliographic details. Pick one canonical source and make all fields
match it — either replace arxiv_id with the correct identifier/DOI for Hawking
1975 and keep title/authors/year/section as-is (update "arxiv_id" to the 1975
DOI or arXiv if available), or update title/authors/year/section to reflect the
2023 paper for arXiv:2310.14698; ensure you modify the JSON keys "arxiv_id",
"title", "authors", and "year" in the same object so they consistently represent
the chosen source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
benchmarks/package with schema, loader, runner, and 25 benchmark tasks for evaluating AI physics reasoningDetails
Schema (
benchmarks/schema.py):BenchmarkTask,BenchmarkSuite,Referencedataclasses with enums forDifficulty,TaskType,OutputFormat. Full JSON roundtrip support.Tasks (
benchmarks/tasks/*.json): 25 problems sourced from standard textbooks and papers (Peskin & Schroeder, Misner/Thorne/Wheeler, Pathria, Ashcroft & Mermin, Nielsen & Chuang, Goldstein, etc.). Each task includes problem statement, given information, assumptions, conventions, expected answer, verification hints, and reference citation.Loader (
benchmarks/loader.py): Discovers task files, loads combined suites, filters by subfield/difficulty/type, and provides inventory summary.Runner (
benchmarks/runner.py): Formats task prompts (excluding answers and hints), invokes a caller-provided model function, collectsTaskResultobjects with timing and error handling, and generates summary reports.Closes #45
ENG-437
Test plan
tests/test_benchmarks.py)benchmarks/tasks/🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests
Chores