Replicating Apple's LLM benchmarking

[meghsat]

Hey all,

I came across this article: https://machinelearning.apple.com/research/exploring-llms-mlx-m5, where Apple claims to have achieved 2.87 sec TTFT on the MacBook Pro M5-24GB for the GPT-OSS-20B-MXFP4-Q4 model using MLX. However, I can’t seem to replicate those numbers — I’m getting a TTFT of ~8 sec.
Note: None of the models listed in the article are performing as claimed.
Here’s my benchmarking setup:

1. To measure TTFT, I had to modify the mlx_lm/generate.py script. Here’s the PR containing those changes: https://github.com/ml-explore/mlx-lm/pull/633/files
2. Once you add the TTFT logic, please run this code:

import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load(
    "mlx-community/gpt-oss-20b-MXFP4-Q4", # You can replace with any model ID mentioned in the article.
    tokenizer_config={"trust_remote_code": True}
)

mx.eval(model.parameters())

vocab_size = tokenizer.vocab_size
prompt_length = 4096

mx.random.seed(0)

dummy_tokens = mx.random.randint(0, vocab_size, (prompt_length,)).tolist()

tokenizer._eos_token_ids = {}

# warmup
response = generate(
    model, 
    tokenizer, 
    prompt=dummy_tokens, 
    max_tokens=128, 
    verbose=True,
    prefill_step_size=4096 
)

# Actual run
response = generate(
    model, 
    tokenizer, 
    prompt=dummy_tokens, 
    max_tokens=128, 
    verbose=True,
    prefill_step_size=4096  
)

It would be great if anyone has observed similar or different results and could share their setup here. Thanks in advance.

[awni]

What's your hardware / OS? To reproduce those numbers you need the latest MLX (0.30.0) on macOS 26.2 (beta release) on the M5.

[meghsat]

Thank you for pointing it out. I'm on macOS 26.1.

[meghsat]

I am able to reproduce the numbers using MacOS 26.2. Thanks!