Skip to content

fix(benchmark): pyright and ruff cache clearing targets wrong paths #2

@w-martin

Description

@w-martin

Problem

benchmarks/benchmark_checkers.py clears these directories before each timed run to simulate a cold-cache analysis:

cache_dirs = [
    target_dir / ".mypy_cache",
    target_dir / ".pyright",
    target_dir / ".ruff_cache",
    target_dir / ".ty",
]

Two of these paths are incorrect, meaning the benchmark does not actually get a cold cache for those tools.

pyright

pyright writes its analysis cache to ~/.cache/pyright/ (macOS/Linux), not to .pyright/ within the project directory. .pyright/ in the project is only pyright's config/exclusion tracking. The timed runs therefore benefit from a warm cache, systematically underestimating pyright's cold-start time.

ruff

ruff's cache lives at ~/.cache/ruff/ (the XDG platform cache dir), not .ruff_cache/ in the project. The correct approach is to pass --no-cache to the ruff invocation rather than deleting a local directory that may not exist.

Fix

  1. pyright: add Path.home() / ".cache" / "pyright" to cache_dirs
  2. ruff: remove .ruff_cache from cache_dirs; add --no-cache to the ruff command in the ToolInfo definition
  3. ty: verify whether ty's cache is in the project dir or in the platform cache dir for the pinned version — adjust accordingly
  4. After correcting, re-run benchmarks; pyright numbers in particular may change

Additional issue: npx pyright startup overhead

The pyright invocation goes through npx, adding ~100–300ms of Node.js/npm resolution overhead to every timed run. If pyright is installed globally (npm install -g pyright) the direct binary invocation should be used instead to measure only pyright's analysis time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions