Skip to content

Add TorsionNet500CCSDT benchmark visualisation, MAE metric, and docs#682

Open
ellalbolland wants to merge 4 commits into
ddmms:mainfrom
ellalbolland:add-torsionnet-mace-benchmark
Open

Add TorsionNet500CCSDT benchmark visualisation, MAE metric, and docs#682
ellalbolland wants to merge 4 commits into
ddmms:mainfrom
ellalbolland:add-torsionnet-mace-benchmark

Conversation

@ellalbolland

Copy link
Copy Markdown

Pre-review checklist for PR author

Summary

Adds per-fragment RMSE/MAE scatter plots and per-fragment torsion energy profile curves to the TorsionNet500CCSDT analysis, enabling table-cell → scatter → curve click-through navigation.

Named TorsionNet500CCSDT rather than TorsionNet500 to coexist alongside the existing barrier-height implementation in #642/#659, since this benchmark reports RMSE/MAE of the full relative energy profile against CCSD(T)/CBS reference data - a different metric and reference level of theory, not a duplicate.

Linked issue

Resolves #679

Progress

  • Calculations
  • Analysis
  • Application
  • Documentation

Testing

Tested against 16 of the 21 registered models (full 500/500-fragment recompute completed for each). The uma family (uma-s-1p1-omat, uma-m-1p1-omat, uma-s-1p1-omol, uma-s-1p2-omol, uma-m-1p1-omol) could not be run due to lacking HuggingFace authorisation for the gated UMA model weights.
Confirmed working end-to-end by running the app standalone (uv run python ml_peg/app/conformers/TorsionNet500CCSDT/app_TorsionNet500CCSDT.py) and clicking through the table -> scatter -> curve interactivity manually.

New decorators/callbacks

One new local callback, _register_curve_callback in app_TorsionNet500CCSDT.py. plot_from_scatter pre-loads every possible figure upfront (too expensive for 500 curves/model); struct_from_scatter lazy-loads but only for structures, not Plotly figures. This one lazy-loads a curve JSON from file per click instead. Flagged with a TODO to move it into build_callbacks.py if another benchmark needs the same pattern.

Adds per-fragment RMSE/MAE scatter plots and per-fragment torsion energy
profile curves to the TorsionNet500CCSDT analysis, wired into the app via a
table-cell -> scatter -> curve click-through. Renames the raw energy fields
to ref_energy/model_energy (and ref_rel_energy/model_rel_energy for the
mean-centered profiles) to match existing codebase conventions, and switches
to the shared rmse/mae utilities instead of a local reimplementation.

Named TorsionNet500CCSDT rather than TorsionNet500 to coexist alongside the
existing barrier-height implementation in ddmms#642/ddmms#659, since this benchmark
reports RMSE/MAE of the full relative energy profile against CCSD(T)/CBS
reference data - a different metric and reference level of theory, not a
duplicate.
Comment thread docs/source/user_guide/benchmarks/conformers.rst
@joehart2001

joehart2001 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Thanks @ellalbolland, from a brief scan its looking great! For the uma model access you can request it i think here: https://huggingface.co/facebook/UMA
it would be good to test. i think you then also need to install the hugging face cli etc. but that should be straightforward

  • please could you also send us the input files for the calc so we can upload them and test it
  • also add the element filtering too

@ElliottKasoar ElliottKasoar added the new benchmark Proposals and suggestions for new benchmarks label Jul 9, 2026
Save per-fragment elemental info via get_struct_info so the app can filter TorsionNet500CCSDT by element, and set precision="high" explicitly on the calculator.
@ellalbolland

Copy link
Copy Markdown
Author

Here's the input dataset used for the calc: TorsionNet500_xyz (1).zip

@joehart2001

Copy link
Copy Markdown
Collaborator

Here's the input dataset used for the calc: TorsionNet500_xyz (1).zip

thanks uploaded

Comment thread ml_peg/calcs/conformers/TorsionNet500CCSDT/calc_TorsionNet500CCSDT.py Outdated
Computational cost
------------------

Medium: tests are likely to take hours per model to run on CPU, since each

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this takes about 10 minutes for me locally on cpu, so lets change to less than an hour on cpu and minutes on GPU

@joehart2001

Copy link
Copy Markdown
Collaborator

Hi @ellalbolland, everything is looking really nice so far!

One thing that would be great to add is visualisaiton from the torsion curve plot. you could take a look at some of the NEB curves to see how they use the trajectory mode to visualise data points along the curve. thanks!

ellalbolland and others added 2 commits July 13, 2026 14:28
…CSDT.py

Co-authored-by: Joseph Hart <92541539+joehart2001@users.noreply.github.com>
…CSDT.py

Co-authored-by: Joseph Hart <92541539+joehart2001@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new benchmark Proposals and suggestions for new benchmarks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TorsionNet500CCSDT benchmark

3 participants