Performance Benchmarks

Model Size and Benchmarks
- Latest
  - Model Sizes
  - Speed Benchmarks

Our design philosophy is to provide high quality compact models balancing between capacity and adequate quality.

It is widely known (also confirmed by our own research) that you can get incremental improvements by scaling your model further 2x, 5x, 10x. But we believe firmly that performance gains should be achieved on similar or lower computation budget.

There are novel techniques that might enable packing models close to EE performance into packages as small as 10-20 MB, but for now the lowest we could achieve was about 50 MB for EE models.

Model Size and Benchmarks

Latest

Model Sizes

Model	Params, M	Model size CE, MB	Model size EE, MB
EN V1	45.6	182	~45.5 (EE)
DE V1	45.6	182	~45.5 (EE)
ES V1	52.8	211	~52.75 (EE)

Speed Benchmarks

It is customary to publish Multiply-Adds or FLOPS as a measure of compute required, but we prefer just sharing model sizes and tests on commodity hardware.

All of the below benchmarks and estimates were run on 6 cores (12 threads) of AMD Ryzen Threadripper 1920X 12-Core Processor (3500 МHz). Scale accordingly for your device. These tests are just run as is using native PyTorch, without any clever batching / concurrency techniques. You are welcome to submit your test results!

Test procedure:

We take 100 10-second audio files
Split into batches of 1, 5, 10, 25 files
Measure how long it takes to process a batch of a given size on CPU
On GPU our models are so fast that batch size and audio length do no really matter (in practical cases)
We measure how many seconds of audio per second one processor core can process. This is similer to 1 / RTF per core

We report results for the following types of models: