Investigate Mac CPU performance gaps in views, permutations, FFT, and small-shape workloads

[terasakisatoshi]

Summary

The latest native macOS CPU benchmark run shows several repeatable performance gaps for tenferro-rs on an Apple M4. This is a consolidated report so the individual regressions can be triaged together.

Benchmark reports:

• CPU ops
• Public API
• Einsum
• Permutation
• FFT
• Linalg JVP/VJP

Environment

• Hardware: Apple M4, macOS 26.5.2, 1 or 4 threads
• tenferro-rs commit: 11a5b5a3c30b6258919557b4b69b429b6e686d75
• CPU backend: system-accelerate, Apple Accelerate
• PyTorch: 2.12.0, Accelerate
• JAX: 0.10.1, XLA CPU
• Timings are medians in milliseconds; inputs/setup are outside the timed region where documented.

Collection command:

PERMUTATION_EXTRA_FEATURES=hptt BENCHMARK_TARGET_PROFILE=mac-cpu ./scripts/run_all.sh 1 4

Observed gaps

1. Small matrix and AD overhead

At 4 threads, tenferro-rs eager mode is substantially slower than PyTorch for small shapes:

benchmark	shape	tenferro eager	PyTorch	ratio
einsum_ij_jk_ik	2x2	0.425 ms	0.029 ms	14.7x
matmul	2x2	0.296 ms	0.010 ms	29.6x
svd	2x2	0.276 ms	0.010 ms	27.6x
solve	2x2, rhs=1	0.332 ms	0.018 ms	18.4x
eigh	2x2	0.231 ms	0.011 ms	21.0x

The traced path reduces the overhead but remains slower for several rows, for example 0.038 ms vs 0.010 ms for 2x2 matmul.

Small reverse-mode AD also shows a large fixed cost:

benchmark	shape	tenferro trace	PyTorch	ratio
grad_sum_qr_vjp	2x2	0.335 ms	0.075 ms	4.5x
grad_sum_qr_jvp	4x4	0.275 ms	0.068 ms	4.0x
grad_sum_eigh_vjp	2x2	0.152 ms	0.041 ms	3.7x

2. Indexing and metadata-only views

The public API report has unusually large gaps:

• gather, f64, 262144 elements, 1 thread: tenferro direct 3.290 ms vs PyTorch 0.180 ms and JAX 0.066 ms.
• gather, 4 threads: tenferro direct 1.060 ms vs PyTorch 0.180 ms and JAX 0.079 ms.
• reshape_view, 33554432 -> 8192x4096: tenferro direct 9.089 ms (1 thread) / 7.928 ms (4 threads), while PyTorch reports 0.001 ms.
• slice_view, 4194304 -> 2096128: tenferro 0.741 / 0.693 ms vs PyTorch 0.001 ms.
• transpose_view, 4096x4096: tenferro 3.283 / 2.944 ms vs PyTorch 0.001 ms.

The view rows are intended to be metadata-only operations. Please verify whether the tenferro public API is materializing or otherwise touching the full storage.

3. Permutation materialization

The largest end-to-end materialization gaps are:

• rotation_6d_32_32_32_32_16_16, 1 thread: tenferro 339.853 ms vs HPTT 217.848 ms and Strided.jl 206.587 ms.
• The same row, 4 threads: tenferro 253.926 ms vs HPTT 117.239 ms.
• reverse_23d_2, 1 thread: tenferro 45.276 ms vs strided-rs 13.166 ms.
• reverse_23d_2, 4 threads: tenferro 21.777 ms vs strided-rs 7.316 ms.
• transpose_3d_256_201, 1 thread: tenferro 25.293 ms vs strided-rs 14.558 ms.
• transpose_3d_256_201, 4 threads: tenferro 8.601 ms vs HPTT 7.572 ms.

These rows measure transpose_view followed by to_contiguous, with destination allocation included and the view construction outside timing.

4. Real FFT

The cached FftExecutor path, which is the primary fair comparison against warmed torch.fft, is slower for real transforms:

operation	dtype	threads	tenferro cached	PyTorch	ratio
rfft	f64	1	7.566 ms	4.885 ms	1.55x
rfft	f64	4	8.070 ms	4.864 ms	1.66x
irfft	c64	1	8.508 ms	5.014 ms	1.70x
irfft	c64	4	7.481 ms	4.965 ms	1.51x

Complex fft/ifft rows do not show the same pattern consistently, so this may be specific to the real-transform implementation or Accelerate dispatch.

5. Eager einsum path overhead

For larger contraction workloads, trace mode is often close to PyTorch, but eager mode adds a measurable overhead:

• str_mps_varying_inner_product_200, 4 threads: eager 19.314 ms vs PyTorch 6.399 ms; trace 8.211 ms.
• str_matrix_chain_multiplication_100, 4 threads: eager 6.777 ms vs PyTorch 3.147 ms; trace 3.397 ms.
• lm_batch_likelihood_sentence_3_12d, 4 threads: eager 40.695 ms vs PyTorch 26.058 ms; trace 16.057 ms.
• lm_batch_likelihood_sentence_4_4d, 4 threads: eager 20.187 ms vs PyTorch 8.418 ms; trace 9.405 ms.

Questions

1. Can the small-shape overhead be separated into tensor allocation, runtime dispatch, layout conversion, and backend call overhead?
2. Do the view APIs currently materialize data on the Accelerate backend?
3. Is the permutation kernel expected to use a specialized blocked/parallel copy path for these stride patterns?
4. Are rfft/irfft using the optimal Accelerate/vDSP path, and is the cached plan actually reused?
5. Could the eager einsum path cache intermediate metadata/plans similarly to the traced path?

The benchmark report contains the raw-run metadata and per-backend logs for follow-up profiling.

[shinaoka]

Remeasurement on current tenferro-rs and root-cause analysis

I migrated the benchmark runners (tenferro-benchmark) to the current tenferro-rs API and re-ran the CPU suite on this machine (Apple M5 Max, macOS, system-accelerate, tenferro-rs 7f91eb7d, benchmark repo commit 9c4ca7d). Two findings substantially change the picture from the original report.

1. The eager-mode small-shape gap was largely a measurement artifact

Two methodology issues inflated eager timings:

• The eager benchmark rebuilt EagerRuntime + CpuBackend on every timed call. A 4-thread CpuBackend construction includes Rayon pool setup, so every small op paid a pool-build cost inside the timed region. The runner now shares one eager runtime via a process-wide OnceLock.
• Input construction was inside the timed region for eager and PyTorch but outside for trace. Data generation, Tensor conversion, and eager metadata registration were measured for eager/PyTorch but not for trace. All arms now split setup (untimed) from the timed execution closure.

With those fixes, 4-thread eager 2x2 numbers vs the original report:

benchmark	issue eager	now eager	now PyTorch
einsum_ij_jk_ik	425 us	6.7 us	4.8 us
matmul	296 us	5.8 us	0.9 us
svd	276 us	26.2 us	3.7 us
solve	332 us	63.9 us	5.3 us
eigh	231 us	12.7 us	2.7 us

2. Root cause of the remaining O(10 us) eager linalg overhead: per-call Rayon install

The remaining linalg gap (svd/solve/eigh at 4 threads) is not LAPACK cost. A direct CpuBackend::svd call is 1.2-1.9 us at 4 threads, and a direct lu_factor+lu_solve is 2.5-3.0 us. The extra cost appears in the eager dispatch path:

• ExtensionExecutor::execute_reads calls to_contiguous_read per input.
• CpuExecSession::to_contiguous_read routes through run_native, which for the BLAS provider executes self.ctx.install(|| op(buffers)) on every call, even for already-compact tensors that only need an Arc clone.
• Rayon install on a 4-thread pool costs ~6 us; solve performs 4 to_contiguous_read calls (LuSolvePrepared has 4 inputs), adding ~24 us.

Isolating the env vars confirmed this: RAYON_NUM_THREADS=4 reproduces the slowdown (eager svd 14 us), while VECLIB_MAXIMUM_THREADS=4 with RAYON_NUM_THREADS=1 stays fast (1.7 us).

Proposed fix (validated in the benchmark checkout's extern tenferro-rs): skip the Rayon install in to_contiguous_read for compact TensorRead::Tensor inputs (the only work is an Arc clone; views still go through the install path for strided kernels). Result at 4 threads:

op (2x2)	before fix	after fix	PyTorch
eager svd	14.2 us	3.3 us	3.7 us
eager solve	42.2 us	5.5 us	5.3 us
eager eigh	8.4 us	2.1 us	2.7 us

All 328 tenferro-cpu unit tests pass with the change. I plan to open a PR against tenferro-rs with this fix, then update the benchmark results and this issue.

Remaining question

matmul 2x2 at 4 threads is still 5.8 us vs 0.9 us for PyTorch; the same to_contiguous path may not explain it (matmul inputs are already compact and the eager dispatch there may hit a different cost). I can dig into that next if useful.

[shinaoka]

Update: root cause fixed, benchmark results updated

The O(10 us) eager linalg overhead at 4 threads is fixed by #1662 (perf(cpu): skip Rayon pool entry for compact to_contiguous_read).

Root cause

CpuExecSession::to_contiguous_read routed every call through the native execution entry (Rayon pool install on the BLAS provider) even for already-compact inputs whose only work is an Arc clone. Rayon install on a 4-thread pool costs ~6 us; solve calls to_contiguous_read once per input (4 inputs for LuSolvePrepared), adding ~24 us per eager call. Env isolation confirmed RAYON_NUM_THREADS=4 reproduces the slowdown while VECLIB_MAXIMUM_THREADS=4 does not.

Remeasured (macOS M5 Max, system-accelerate, tenferro-rs 3ce8fbed = v0.2.0 + fix)

Eager mode, 4 threads, medians in us:

benchmark (2x2)	original issue	before fix	after fix	PyTorch
einsum_ij_jk_ik	425	6.7	4.8	4.9
matmul	296	5.8	6.2	0.9
svd	276	26.2	5.3	3.6
solve	332	63.9	7.9	5.2
eigh	231	12.7	3.7	2.7
qr	—	14.0	3.8	13.7

svd/solve/eigh/qr are now within 1.5x of PyTorch (qr is faster than PyTorch). matmul 2x2 remains ~7x slower than PyTorch; that is a separate dispatch cost (not the to_contiguous_read path, since the fix does not change it) and is still under investigation.

Benchmark runner changes (API migration, shared eager runtime, input construction outside the timed region, run_all.sh 1 4) are committed in tenferro-benchmark (9c4ca7d, results f0c8ec4).