HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images

Yumeng Liu¹, Xiao-Xiao Long², Marc Habermann³, Xuanze Yang¹, Cheng Lin⁴,
Yuan Liu⁵, Yuexin Ma⁶, Ligang Liu^1*

¹USTC   ²Nanjing University   ³MPI-INF   ⁴MUST Macau   ⁵HKUST   ⁶ShanghaiTech  
^*Corresponding author

---

We introduce Hand Geometry Grounding Transformer (HGGT), a scalable and generalized solution for 3D hand mesh recovery. Our method unifies diverse data sources to achieve robust performance across varying camera viewpoints and environments.

---

Table of Contents

• ⚙️ Installation
• 🚀 Demo (pre-cropped multi-view images)
• 📊 Evaluation
• 📁 Dataset
• 🙏 Acknowledgements

---

TL;DR

We present the first feed-forward framework that jointly estimates 3D hand meshes and camera poses from uncalibrated multi-view images.

TODO

• Release synthetic dataset on Hugging Face
• Release dataset generation pipeline code (due July 17)
• Release pretrained model checkpoints
• Release model inference code (due July 17)
• Release evaluation scripts (due July 20)

---

⚙️ Installation

Create the conda environment

conda create -n hggt python=3.10 -y
conda activate hggt

pip install -r requirements.txt

MANO models (required for mesh overlay and evaluation)

Download MANO from the MANO website, unzip, and place the model files under:

assets/mano_v1_2/models/MANO_RIGHT.pkl

Mesh overlays and multi-view image evaluation require MANO; loading the network and writing mano_params to result.npz does not.

Pretrained weights

Default demo loading uses Hugging Face:

https://huggingface.co/catmint123/HGGT

via HGGT.from_pretrained("catmint123/HGGT"). A local training checkpoint (.pt with a model state dict) can be passed with --checkpoint.

---

🚀 Demo (pre-cropped multi-view images)

This demo reads already hand-cropped multi-view images (one square crop per view), runs HGGT, and writes a mosaic / optional mesh overlays / result.npz.

Hand detection from full-frame images and video demos will be added in a follow-up release.

# Example: Arctic sample (multi-view)
python demo/demo_multiview_images.py \
  --image_folder examples/multiview/Arctic/sample_0000 \
  --output_dir outputs/demo_arctic_0000

# Optional: local checkpoint and MANO path
python demo/demo_multiview_images.py \
  --image_folder examples/multiview/HO3D/sample_0000 \
  --checkpoint /path/to/checkpoint.pt \
  --mano_model_path assets/mano_v1_2/models \
  --output_dir outputs/demo_ho3d_0000

Bundled examples live under examples/multiview/ (2 samples from each of HO3D, DexYCB, Arctic, Interhand, Oakink, Freihand). Freihand examples are single-view.

Typical outputs:

• input_mosaic.jpg — input views side-by-side
• overlay_mosaic.jpg — solid mesh overlays for all views side-by-side (pyrender)
• result.npz — mano_params, cameras, and vertices when MANO is available

---

📊 Evaluation

Evaluate HGGT on multi-view WebDataset tars from standard hand datasets (HO3D, DexYCB, Arctic, Interhand, Oakink, Freihand).

Download the evaluation data from JubSteven/POEM-v2. Full setup, launchers (run_eval_mv_image.sh, eval_mv_image.slurm), and metric details are in eval/README.md.

---

📁 Dataset

Download

Our synthetic dataset is available on Hugging Face:

# Download via huggingface_hub
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='catmint123/HGGT-synthetic-data',
    repo_type='dataset',
    local_dir='data/hggt_synthetic',
)
"

After downloading, extract the tar shards:

cd data/hggt_synthetic/small
for f in *.tar; do tar -xf "$f"; done

Please refer to the dataset page for details on the dataset structure.

---

🙏 Acknowledgements

We would like to express our gratitude to the authors and contributors of the following projects:

• VGGT
• GraspXL
• POEM-v2

---

Citation

If you find our work useful, please consider citing us using the following BibTeX entry:

@article{liu2026hggt,
  title={HGGT: Robust and Flexible 3D Hand Mesh Reconstruction from Uncalibrated Images},
  author={Liu, Yumeng and Long, Xiao-Xiao and Habermann, Marc and Yang, Xuanze and Lin, Cheng and Liu, Yuan and Ma, Yuexin and Liu, Ligang},
  journal={arXiv preprint arXiv:2603.23997},
  year={2026}
}

---

License

This project is licensed under the Apache License 2.0.