Measuring prefill throughput with Qwen3-4B-2507-int4 on A770

[SearchSavior]

Hello,

I am trying to leverage perf_metrics to obtain granular performance data on prefill throughput with LLM.

Section 2.2 of this paper has discussion which suggests we should take input_token / ttft to measure latency before decode phase begins. I am wondering if this is correct and if it would make sense as a feature.

[as-suvorov]

Hello,

perf_metrics reports num_input_tokens and ttft already. Prefill throughput can be calculated by dividing. What's your proposal? To include prefill_throughput into perf_metrics?
cc: @pavel-esir

[SearchSavior]

Yes. My intention for including '''prefill_throughput''' was to have greater parity with llama.cpp for a solid bench against Vulkan, ipex etc using my project.

Other inference engines provide some form of calculated prefill_throughput so I think a default would be useful.

Would be interested to contribute this feature. Have been building with OpenVINO for over a year now and this seems like an approachable first contribution.

Can you give me some direction? As a problem statement, add calculated prefill_throughput to perf_metrics seems concrete.

[apaniukov]

Hi @SearchSavior,
The easiest path is to add the get_prefill_throughput method to the PerfMetrics struct.
For a single run metric (one generate call), the calculation is straightforward. If you are accumulating the results, then the num_input_tokens will contain the sum of each prompt length, while the get_ttft will return a pair of (mean, std) statistics. There are two ways here:

1. The raw_metrics contains ttft for each input; you can divide the num_input_tokens by the sum of raw TTFT
2. Add a new field to metrics that will accumulate TTFT for each .generate in a vector and modify + operator accordingly. This way is more consistent with other metrics, you can calculate the stats and return (mean, std) pair for TTFT as well.

After that, there are two more steps:

1. Update Python bindings according to the changes
2. Update CPP and Python benchmark samples to include the new metric

Let me know if you have any further questions.

[SearchSavior]

@apaniukov I certainly will, thank you for the direction!!