Skip to content

Roadmap

Jump to bottom
Alessandro Cosentino edited this page Jul 24, 2025 · 3 revisions

This is a public roadmap for the overall Metriq project, that is, both metriq-gym and the metriq.info website.

Presentation

  • Documenting a description of each benchmark (and a mechanism to do that)
  • Access control on submissions (particularly important for metriq-gym results)
  • CI/CD on metriq.info website
  • Infrastructure to curate pages on metriq.info (similar to OWID Runner) – metriq-studio proposal

Runner

  • Mechanism to run a suite of tests (as opposed to a single benchmark) – https://github.com/unitaryfoundation/metriq-gym/issues/416
  • Mechanism to run benchmark synchronously (optional no async dispatch/poll)
  • Scheduled / continuous execution
  • Testing on dedicated / on-premise hardware (no Cloud API) – potential partnership can help with this
  • Parallel execution and batching of benchmarks
  • Web frontend (on the side of the existing CLI frontend)
  • Advanced cloud resource management (e.g. shortest queue for a device)

Data

Clone this wiki locally