Repository navigation
How do we obtain evaluation metrics for this demo subset? #23
Description
Hi ML Commons team,
I am looking to evaluate some HuggingFace models on this demo benchmark but I am not sure how to properly evaluate the responses. Is there any documentation that I missed that teaches how to run the benchmark and obtain safety numbers for any given HF dataset?
Thank you!
Activity
bollacker commentedon Mar 14, 2025
A good question. You are welcome to use the code at https://github.com/mlcommons/modelbench to operationalize running the benchmark on any model. The docs aren't great, but you can join our discord ( https://discord.gg/v5K82PsJ ) and ask for help in the #engineering-support channel.
You can use any available public evaluator/guard model to your evaluation (I think at least one LlamaGuard is already implemented). Such existing guard models are not representative of the scores that the internal AILuminate evaluator provides, so you can't get anything resembling an official score. If you have a set of HF-hosted models in mind for official scoring, reach our and perhaps we can add them to our list of benchmarks to run.
Kurt :-)
ljvmiranda921 commentedon Mar 14, 2025
Hi @bollacker , thanks for the reply! Thanks for linking modelbench, will poke around it for a bit :)
Btw, the discord invite link seems expired(NVM, I'm part of the Discord group now). Will reach out if I encountered any issues. Cheers!