Problem
benchmarks/benchmark_checkers.py clears these directories before each timed run to simulate a cold-cache analysis:
cache_dirs = [
target_dir / ".mypy_cache",
target_dir / ".pyright",
target_dir / ".ruff_cache",
target_dir / ".ty",
]
Two of these paths are incorrect, meaning the benchmark does not actually get a cold cache for those tools.
pyright
pyright writes its analysis cache to ~/.cache/pyright/ (macOS/Linux), not to .pyright/ within the project directory. .pyright/ in the project is only pyright's config/exclusion tracking. The timed runs therefore benefit from a warm cache, systematically underestimating pyright's cold-start time.
ruff
ruff's cache lives at ~/.cache/ruff/ (the XDG platform cache dir), not .ruff_cache/ in the project. The correct approach is to pass --no-cache to the ruff invocation rather than deleting a local directory that may not exist.
Fix
- pyright: add
Path.home() / ".cache" / "pyright" to cache_dirs
- ruff: remove
.ruff_cache from cache_dirs; add --no-cache to the ruff command in the ToolInfo definition
- ty: verify whether ty's cache is in the project dir or in the platform cache dir for the pinned version — adjust accordingly
- After correcting, re-run benchmarks; pyright numbers in particular may change
Additional issue: npx pyright startup overhead
The pyright invocation goes through npx, adding ~100–300ms of Node.js/npm resolution overhead to every timed run. If pyright is installed globally (npm install -g pyright) the direct binary invocation should be used instead to measure only pyright's analysis time.
Problem
benchmarks/benchmark_checkers.pyclears these directories before each timed run to simulate a cold-cache analysis:Two of these paths are incorrect, meaning the benchmark does not actually get a cold cache for those tools.
pyright
pyright writes its analysis cache to
~/.cache/pyright/(macOS/Linux), not to.pyright/within the project directory..pyright/in the project is only pyright's config/exclusion tracking. The timed runs therefore benefit from a warm cache, systematically underestimating pyright's cold-start time.ruff
ruff's cache lives at
~/.cache/ruff/(the XDG platform cache dir), not.ruff_cache/in the project. The correct approach is to pass--no-cacheto the ruff invocation rather than deleting a local directory that may not exist.Fix
Path.home() / ".cache" / "pyright"tocache_dirs.ruff_cachefromcache_dirs; add--no-cacheto the ruff command in theToolInfodefinitionAdditional issue:
npx pyrightstartup overheadThe pyright invocation goes through
npx, adding ~100–300ms of Node.js/npm resolution overhead to every timed run. If pyright is installed globally (npm install -g pyright) the direct binary invocation should be used instead to measure only pyright's analysis time.