Conversation
- Reject eval.py wrapping; use make_env() + existing evaluate_policy() - Add PolicyAdapter Protocol and lerobot_policy_adapter.py - Extract shared lerobot_env.py from g1_native example - Keep generic API intact; add evaluate_lerobot_policy() as additive layer - Broaden policy interface to dict+array observations - Add 3 new test modules and action shape validation - Add torch to [lerobot] optional deps - Include ASCII data flow diagram and failure mode table Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add PolicyAdapter protocol and LeRobotPolicyAdapter for checkpoint loading - Extract shared LeRobot env utilities into lerobot_env.py - Add evaluate_lerobot_policy() entry point with action shape validation - Update examples/lerobot_eval_harness.py with --checkpoint-path and --repo-id CLI - Add tests for policy adapter, env creation, eval plugin, and harness example - Update README with LeRobot CI evaluation section - Add torch/lerobot to mypy ignore_missing_imports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements the LeRobot evaluation plugin as planned in
.planning/lerobot-eval-plugin/PLAN.md.What's new
PolicyAdapterprotocol +LeRobotPolicyAdapterfor loading real LeRobot checkpointslerobot_env.pywithcreate_native_env(), headless config patching, and MuJoCo rendering injectionevaluate_lerobot_policy()entry point with action shape validation and repo_id inferenceexamples/lerobot_eval_harness.pywith--checkpoint-path,--repo-id, and--assert-thresholdCLIVerification
pytest: 495 passed, 9 skipped, 94.11% coverageruff check .: cleanmypy src/: clean🤖 Generated with Claude Code