Any interest in adding ML Perf Llama 3 8B to TorchTitan models ?

[githubsgi]

It will be great to have MLPerf LLama 3 pre-training working OOB with TorchTitan, Here are some references on that .

MLPerf Training Adds Llama 3.1 8B Benchmark

small_llm_pretraining/nemo

[githubsgi]

@tianyu-l , @wconstab , any thoughts on this ? Having such a model OOB would greatly facilitates quick validation and comparison or full HW and SW stack.

[tianyu-l]

Having such a model OOB would greatly facilitates quick validation and comparison or full HW and SW stack.

sorry, could you tell a more complete story? Not clear to me what's going on.

[githubsgi]

@tianyu-l , a quick summary of MLPerf follows.

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. MLPerf is an industry-standard benchmark suite from MLCommons for measuring the performance of machine learning (ML) hardware, software, and cloud platforms. It provides standardized, apples-to-apples comparisons across different ML tasks, covering both the training of AI models and the inference (or running) of those models on a wide range of systems, from data centers to mobile devices. The benchmarks are a result of a consortium of AI industry leaders, researchers, and academics.

MLPerf Training v5.1 introduces a benchmark based on Meta’s Llama 3.1 8B, replacing BERT with a modern model that can still run on single-node systems. This new pretraining benchmark based on Meta’s Llama 3.1 8B model brings modern LLM pretraining evaluation within reach of more organizations. It was intentionally designed to be easy to set up and run on small to moderately sized computational resources. By requiring only a single node, using a subset of the C4 dataset, and starting from random weights, it lowers the barrier to entry while maintaining relevance to current AI development practices.

[githubsgi]

@tianyu-l , does that give you enough context about MLPerf ? I am only proposing support for Llama3 8B MLPerf pre-training case - not any of the other MLPerf case at this point.

[tianyu-l]

I still don't know what's the point of supporting MLPerf version of Llama3 8B, how's it different from current Llama 3 8B. Did you mention it anywhere?