Skip to content

README.md

Latest commit

 

History

History
89 lines (66 loc) · 3.37 KB

README.md

File metadata and controls

89 lines (66 loc) · 3.37 KB
Raw

Finetuning SO100 Model

This guide shows how to finetune dataset collected from SO100 robot, and evaluate the model on the real robot.

Dataset

To collect the dataset via teleoperation, please refer to the official documentation in lerobot: https://huggingface.co/docs/lerobot/il_robots?teleoperate_so101=Command

Dataset Path: izuluaga/finish_sandwich

Visualize it with this link

Handling the dataset

uv run --project scripts/lerobot_conversion \
  python scripts/lerobot_conversion/convert_v3_to_v2.py \
  --repo-id izuluaga/finish_sandwich \
  --root examples/SO100/finish_sandwich_lerobot

Then move the modality.json file to the root of the dataset.

cp examples/SO100/modality.json examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich/meta/modality.json

Finetuning

Run the shared finetune launcher directly, using absolute joint positions (feel free to experiment with relative positions):

CUDA_VISIBLE_DEVICES=0 NUM_GPUS=1 uv run bash examples/finetune.sh \
  --base-model-path nvidia/GR00T-N1.7-3B \
  --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich \
  --modality-config-path examples/SO100/so100_config.py \
  --embodiment-tag NEW_EMBODIMENT \
  --output-dir /tmp/so100_finetune

Open-Loop Evaluation

Evaluate the finetuned model with the following command:

uv run python gr00t/eval/open_loop_eval.py \
  --dataset-path examples/SO100/finish_sandwich_lerobot/izuluaga/finish_sandwich/ \
  --embodiment-tag NEW_EMBODIMENT \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --traj-ids 0 \
  --action-horizon 16 \
  --steps 400

Evaluation Results

The evaluation produces visualizations comparing predicted actions against ground truth trajectories:

Open-loop evaluation results showing predicted vs ground truth trajectories

To read these numbers and decide whether your fine-tune is working, see Interpreting the Result: Is My Fine-tune Working?.

Closed-Loop Evaluation

Please refer to eval_so100.py for how to write SO100 deployment code using Policy API.

  1. set up client side deps
cd gr00t/eval/real_robot/SO100
uv venv
source .venv/bin/activate
uv pip install -e . --verbose
uv pip install --no-deps -e ../../../../
  1. Start policy server from the repository root in a separate terminal:
uv run python gr00t/eval/run_gr00t_server.py \
  --model-path /tmp/so100_finetune/checkpoint-10000 \
  --embodiment-tag NEW_EMBODIMENT
  1. Run the eval script as the client from the gr00t/eval/real_robot/SO100 environment created above:
uv run python eval_so100.py \
  --robot.type=so101_follower --robot.port=/dev/ttyACM2 \
  --robot.id=orange_follower \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}, front: {type: opencv, index_or_path: 6, width: 640, height: 480, fps: 30}}" \
  --policy-host=localhost --policy-port=5555 --lang-instruction="finish the ham cheese olives sandwich"