Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 143 additions & 0 deletions codemods/debarrel/benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
# debarrel benchmark

A small, dependency-free shell script to measure **build** and **test** wall-clock
time of a target repository, so you can quantify the impact of the `debarrel`
codemod.

The intended workflow is:

1. Measure a **baseline** on the untouched repo.
2. Run the `debarrel` codemod on that repo.
3. Measure again (**debarreled**).
4. Diff the two result files.

## Requirements

- `bash`, `python3`, and `/usr/bin/time` (preinstalled on macOS and most Linux distros)
- Whatever toolchain the target repo needs to build and test (e.g. `yarn`)
- The target repo already has its dependencies installed (`yarn install`, etc.)

No packages are installed by the script itself.

## Inputs

Everything is configurable via flags or environment variables. Flags win over
env vars.

| Flag | Env var | Default | Description |
|------|---------|---------|-------------|
| `--repo <path>` | `TARGET_REPO` | `$PWD` | Repository to benchmark |
| `[label]` / `--label <name>` | `LABEL` | `baseline` | Label for this run (used in output filenames) |
| `--build-cmd <cmd>` | `BUILD_CMD` | `yarn build --force` | Command used to build |
| `--test-cmd <cmd>` | `TEST_CMD` | `yarn test` | Command used to test |
| `--clean-cmd <cmd>` | `CLEAN_CMD` | `rm -rf apps/web/.next` | Runs before **each** build run and once before the test phase, for a cold measurement |
| `--build-runs <n>` | `BUILD_RUNS` | `3` | Number of build repetitions |
| `--test-runs <n>` | `TEST_RUNS` | `3` | Number of test repetitions |
| `--out <dir>` | `OUTDIR` | `<repo>/bench-results` | Where to write results |
| `--no-build` | `DO_BUILD=0` | — | Skip the build phase |
| `--no-test` | `DO_TEST=0` | — | Skip the test phase |
| `-h`, `--help` | — | — | Print usage |

> The defaults target the [cal.com](https://github.com/calcom/cal.com) monorepo.
> For other repos, override `--build-cmd`, `--test-cmd`, and `--clean-cmd`.

### Why a "clean" command?

Build tools cache aggressively (Turbo, Next.js, etc.). To measure the *real*
build cost — which is what debarreling changes — each build run first runs the
clean command and the build uses a cache-busting flag (`--force` for Turbo).

The clean command also runs once before the test phase: a leftover build
directory (like `apps/web/.next`) can make the test runner pick up compiled
`*.test.js` files, which adds noise and can cause false failures.

## Outputs

Two files per invocation, written to the output directory:

- `<timestamp>_<label>_<sha>.md` — human-readable summary with per-run times
and `min / mean / max` for each phase.
- `<timestamp>_<label>_<sha>.log` — full stdout/stderr of every command, for
debugging failures.

Example summary:

```md
# Benchmark: baseline

- Date: Fri Jul 17 01:20:00 PDT 2026
- Repo: `/Users/you/cal.com`
- Git: `f004349273` on `main`
- Node: v20.17.0
- Build cmd: `yarn build --force` (runs: 3)
- Test cmd: `yarn test` (runs: 3)
- Clean cmd: `rm -rf apps/web/.next`

## Build (`yarn build --force`)

- run 1: 74.35s (exit 0)
- run 2: 72.10s (exit 0)
- run 3: 73.02s (exit 0)

**Build: min=72.10s mean=73.16s max=74.35s**

## Test (`yarn test`)

- run 1: 27.01s (exit 0)
- run 2: 26.44s (exit 0)
- run 3: 26.88s (exit 0)

**Test: min=26.44s mean=26.78s max=27.01s**
```

## How to run

From this directory (or reference it by full path):

```bash
# 1) Baseline — before running the codemod
./bench.sh baseline --repo /path/to/cal.com

# 2) Run the debarrel codemod on the target repo
# (from the target repo)
codemod run debarrel

# 3) Debarreled — after the codemod
./bench.sh debarreled --repo /path/to/cal.com
```

Custom commands for a non-cal.com repo:

```bash
./bench.sh baseline \
--repo /path/to/app \
--build-cmd "pnpm build" \
--test-cmd "pnpm test" \
--clean-cmd "rm -rf dist .turbo" \
--build-runs 5 --test-runs 5
```

Only benchmark tests (skip the slow build phase):

```bash
./bench.sh baseline --repo /path/to/app --no-build
```

## Comparing results

The `min` (fastest, least-noisy) run is usually the most reliable single number
to compare. Diff the two summary files:

```bash
diff \
bench-results/<baseline>.md \
bench-results/<debarreled>.md
```

## Notes on accuracy

- Close other heavy apps; wall-clock time is sensitive to CPU/thermal load.
- Stop any running dev server for the target repo before benchmarking to avoid
file/CPU contention.
- Run on power (not battery) on laptops.
- Prefer comparing `min` across runs; use `mean`/`max` to judge variance.
160 changes: 160 additions & 0 deletions codemods/debarrel/benchmark/bench.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
#!/usr/bin/env bash
#
# Benchmark build & test wall-clock time for a target repository.
#
# Intended workflow: run this once BEFORE applying the debarrel codemod
# (label "baseline") and once AFTER (label "debarreled"), then diff the two
# result files to see how removing barrel files affected build/test time.
#
# The script is repo-agnostic: point it at any repo and override the build /
# test / clean commands to match that repo's tooling.
#
# ---------------------------------------------------------------------------
# Usage:
# bench.sh [label] [--repo <path>] [options]
#
# Options (all have env-var equivalents in parentheses):
# --repo <path> Repo to benchmark (TARGET_REPO, default: $PWD)
# --label <name> Label for this run (LABEL, default: baseline)
# --build-cmd <cmd> Build command (BUILD_CMD, default: "yarn build --force")
# --test-cmd <cmd> Test command (TEST_CMD, default: "yarn test")
# --clean-cmd <cmd> Cleanup run before each build and once before tests,
# for a cold measurement (CLEAN_CMD, default: "rm -rf apps/web/.next")
# --build-runs <n> Build repetitions (BUILD_RUNS, default: 3)
# --test-runs <n> Test repetitions (TEST_RUNS, default: 3)
# --out <dir> Output directory (OUTDIR, default: <repo>/bench-results)
# --no-build Skip the build phase (DO_BUILD=0)
# --no-test Skip the test phase (DO_TEST=0)
# -h, --help Show this help
#
# Output:
# <out>/<timestamp>_<label>_<sha>.md Markdown summary (min/mean/max)
# <out>/<timestamp>_<label>_<sha>.log Full command output for debugging
#
# The defaults target the cal.com monorepo. For other repos, override
# --build-cmd / --test-cmd / --clean-cmd accordingly (see README.md).
# ---------------------------------------------------------------------------

set -uo pipefail

TARGET_REPO="${TARGET_REPO:-$PWD}"
LABEL="${LABEL:-baseline}"
BUILD_CMD="${BUILD_CMD:-yarn build --force}"
TEST_CMD="${TEST_CMD:-yarn test}"
CLEAN_CMD="${CLEAN_CMD:-rm -rf apps/web/.next}"
BUILD_RUNS="${BUILD_RUNS:-3}"
TEST_RUNS="${TEST_RUNS:-3}"
DO_BUILD="${DO_BUILD:-1}"
DO_TEST="${DO_TEST:-1}"
OUTDIR="${OUTDIR:-}"

show_help() { sed -n '2,45p' "$0" | sed 's/^# \{0,1\}//'; }

while [ $# -gt 0 ]; do
case "$1" in
--repo) TARGET_REPO="$2"; shift 2;;
--label) LABEL="$2"; shift 2;;
--build-cmd) BUILD_CMD="$2"; shift 2;;
--test-cmd) TEST_CMD="$2"; shift 2;;
--clean-cmd) CLEAN_CMD="$2"; shift 2;;
--build-runs) BUILD_RUNS="$2"; shift 2;;
--test-runs) TEST_RUNS="$2"; shift 2;;
--out) OUTDIR="$2"; shift 2;;
--no-build) DO_BUILD=0; shift;;
--no-test) DO_TEST=0; shift;;
-h|--help) show_help; exit 0;;
-*) echo "Unknown option: $1" >&2; exit 2;;
*) LABEL="$1"; shift;;
esac
done

if [ ! -d "$TARGET_REPO" ]; then
echo "Target repo not found: $TARGET_REPO" >&2
exit 1
fi
TARGET_REPO="$(cd "$TARGET_REPO" && pwd)"
cd "$TARGET_REPO"

: "${OUTDIR:=$TARGET_REPO/bench-results}"
mkdir -p "$OUTDIR"

SHA="$(git rev-parse --short HEAD 2>/dev/null || echo nogit)"
BRANCH="$(git branch --show-current 2>/dev/null || echo nogit)"
STAMP="$(date +%Y%m%d-%H%M%S)"
OUT="$OUTDIR/${STAMP}_${LABEL}_${SHA}.md"
LOG="$OUTDIR/${STAMP}_${LABEL}_${SHA}.log"

# Run a command under `/usr/bin/time -p` and echo "<real_seconds> <exit_code>".
# Uses `real` (wall clock) which reflects the full process tree.
time_real() {
local tmp real code
tmp="$(mktemp)"
/usr/bin/time -p bash -c "$1" >>"$LOG" 2>"$tmp"
code=$?
cat "$tmp" >>"$LOG"
real="$(grep '^real' "$tmp" | tail -1 | awk '{print $2}')"
rm -f "$tmp"
echo "${real:-NaN} ${code}"
}

# min / mean / max from a list of numbers.
stats() {
python3 - "$@" <<'PY'
import sys
vals = [float(x) for x in sys.argv[1:] if x not in ("", "NaN")]
if not vals:
print("n/a"); sys.exit(0)
mean = sum(vals) / len(vals)
print(f"min={min(vals):.2f}s mean={mean:.2f}s max={max(vals):.2f}s")
PY
}

{
echo "# Benchmark: $LABEL"
echo
echo "- Date: $(date)"
echo "- Repo: \`$TARGET_REPO\`"
echo "- Git: \`$SHA\` on \`$BRANCH\`"
echo "- Node: $(node --version 2>/dev/null)"
echo "- Build cmd: \`$BUILD_CMD\` (runs: $BUILD_RUNS)"
echo "- Test cmd: \`$TEST_CMD\` (runs: $TEST_RUNS)"
echo "- Clean cmd: \`${CLEAN_CMD:-<none>}\`"
echo
} | tee "$OUT"

if [ "$DO_BUILD" = "1" ]; then
echo "## Build (\`$BUILD_CMD\`)" | tee -a "$OUT"
echo | tee -a "$OUT"
build_times=()
for i in $(seq 1 "$BUILD_RUNS"); do
[ -n "$CLEAN_CMD" ] && eval "$CLEAN_CMD" >>"$LOG" 2>&1
read -r t c < <(time_real "$BUILD_CMD")
echo "- run $i: ${t}s (exit $c)" | tee -a "$OUT"
build_times+=("$t")
done
echo | tee -a "$OUT"
echo "**Build: $(stats "${build_times[@]}")**" | tee -a "$OUT"
echo | tee -a "$OUT"
fi

if [ "$DO_TEST" = "1" ]; then
echo "## Test (\`$TEST_CMD\`)" | tee -a "$OUT"
echo | tee -a "$OUT"
# Clean once so tests run on source, not on stale build output. A leftover
# build dir (e.g. apps/web/.next) makes some runners discover compiled
# *.test.js files, adding noise and false failures.
[ -n "$CLEAN_CMD" ] && eval "$CLEAN_CMD" >>"$LOG" 2>&1
test_times=()
for i in $(seq 1 "$TEST_RUNS"); do
read -r t c < <(time_real "$TEST_CMD")
echo "- run $i: ${t}s (exit $c)" | tee -a "$OUT"
test_times+=("$t")
done
echo | tee -a "$OUT"
echo "**Test: $(stats "${test_times[@]}")**" | tee -a "$OUT"
echo | tee -a "$OUT"
fi

echo "Full command output logged to: $LOG" | tee -a "$OUT"
echo | tee -a "$OUT"
echo "Results written to $OUT"
Loading
Loading