Add cactus bench: CPU throughput benchmarking for Mac and Android#698
Open
ncylich wants to merge 4 commits into
Open
Add cactus bench: CPU throughput benchmarking for Mac and Android#698ncylich wants to merge 4 commits into
ncylich wants to merge 4 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an Android (arm64-v8a) end-to-end and exact-spec benchmark harness under tests/android-e2e/, intended to run on-device via adb and link against engine archives produced by android/build.sh.
Changes:
- Introduces two Android benchmark executables:
e2e_bench(prompt-based) andllm_bench(token-ID exact-spec viacactus_benchmark_tokens). - Adds build + run scripts (
build.sh,run.sh,run_spec.sh) with sharedbench_common.shhelpers for pushing artifacts, disabling cloud/telemetry, and power-state warnings. - Documents a “trustworthy numbers” measurement protocol and known performance pitfalls in a dedicated README.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/android-e2e/run.sh | Runs prompt-based e2e benchmark on-device and pulls per-round CSV output. |
| tests/android-e2e/run_spec.sh | Runs exact-spec token-ID benchmark on-device and prints per-round JSON-derived metrics. |
| tests/android-e2e/README.md | Documents setup, required performance protocol, and reference numbers. |
| tests/android-e2e/e2e_driver.h | Declares the minimal backend/driver interface and CSV/summary helpers. |
| tests/android-e2e/e2e_driver.cpp | Implements scheduling, CSV output, and summary statistics. |
| tests/android-e2e/e2e_bench.cpp | Implements the benchmark runner and minimal config parsing. |
| tests/android-e2e/e2e_backend_cactus.cpp | Implements the Cactus backend adapter for the e2e harness. |
| tests/android-e2e/CMakeLists.txt | Defines Android CMake build for e2e_bench and llm_bench linked to engine archives. |
| tests/android-e2e/cactus_llm_bench.cpp | Implements llm_bench wrapper around cactus_benchmark_tokens. |
| tests/android-e2e/build.sh | Convenience script to build engine archives (optional) and benchmark binaries via CMake. |
| tests/android-e2e/bench_common.sh | Shared adb/push helpers and environment setup for both run scripts. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+4
to
+8
| #include <fstream> | ||
| #include <iostream> | ||
| #include <sstream> | ||
| #include <string> | ||
| #include <vector> |
Comment on lines
+5
to
+9
| #include <chrono> | ||
| #include <cstring> | ||
| #include <iostream> | ||
| #include <string> | ||
|
|
Comment on lines
+35
to
+36
| while (*pos && (*pos == ' ' || *pos == ':' || *pos == '\t')) pos++; | ||
| return atof(pos); |
Comment on lines
+71
to
+75
| char response[8192] = {}; | ||
| auto wall_start = std::chrono::steady_clock::now(); | ||
| int ret = cactus_complete(h->model, messages.c_str(), response, sizeof(response), | ||
| options.c_str(), nullptr, nullptr, nullptr, nullptr, 0); | ||
| auto wall_end = std::chrono::steady_clock::now(); |
Comment on lines
+34
to
+41
| echo "==> Running e2e_bench (max-tokens=$MAX_TOKENS, rounds=$ROUNDS)" | ||
| "${ADB[@]}" shell "cd $DEVICE_DIR && $BENCH_ENV ./e2e_bench \ | ||
| --model-config $DEVICE_DIR/models.json \ | ||
| --backends cactus \ | ||
| --rounds $ROUNDS \ | ||
| --max-tokens $MAX_TOKENS \ | ||
| --prompt \"$PROMPT\" \ | ||
| --output $DEVICE_DIR/e2e_results_android.csv" |
Comment on lines
+17
to
+18
| python3 -c "print(','.join(str(1000+i) for i in range($PREFILL_TOKENS)))" > /tmp/spec_tokens.csv | ||
| "${ADB[@]}" push /tmp/spec_tokens.csv "$DEVICE_DIR/spec_tokens.csv" >/dev/null |
This was referenced Jun 10, 2026
This was referenced Jun 11, 2026
Merged
ncylich
force-pushed
the
android-bench
branch
2 times, most recently
from
June 12, 2026 17:21
ce8844e to
9bc027c
Compare
Token mode prefills exact raw token IDs and times decode via cactus_benchmark_tokens, so prefill numbers carry no tokenizer, template, or chunk-padding variance; prompt mode runs a greedy chat-path completion after a discarded in-process warmup. Registered in the host and Android test builds. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
ncylich
changed the title
Builds the llm_bench fixture through the existing test build trees (host CMake, or the NDK arm64-v8a toolchain for --android), provisions the bundle like cactus test does, and runs a 1-warmup + N-round protocol over the 512+32 token workload or a --prompt chat workload. Android runs push weights once, warn when the device is not charging, and accept a taskset core mask; cloud handoff and telemetry are disabled for every round. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Device preparation, core configuration, chunk-padding interpretation, bundle-vintage checks, and the verified per-device reference table. Signed-off-by: Noah Cylich <noahcylich@gmail.com>
Signed-off-by: Noah Cylich <noahcylich@gmail.com> # Conflicts: # python/cactus/cli/__init__.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
cactus bench, a CLI verb for reproducible CPU inference throughput benchmarks. Runs locally by default;--androidcross-builds for arm64-v8a and runs on a connected device over adb. This replaces the standalonetests/android-e2ebash/C++ harness from earlier revisions of this PR — the protocol is unchanged, but it now rides the existing infrastructure end to end.--prefill-tokensraw token IDs (512 = 4 full 128-token prefill chunks, zero padding) +--decode-tokensdecode viacactus_benchmark_tokens. No tokenizer, template, or padding variance; fresh process per round, first (warmup) round discarded.--prompt): greedycactus_completethrough the chat-template path with a discarded in-process warmup per round; also reports TTFT.CACTUS_DISABLE_CLOUD_HANDOFF=1andCACTUS_NO_CLOUD_TELE=1so no routing probe, network fallback, or telemetry upload lands inside a measurement.--if-missingsemantics), warn when the device is not charging, and accept a--cpu-masktaskset mask for core-restricted runs.How it reuses existing infrastructure
cactus_benchmark_tokens; the only new C++ is a ~120-line driver (cactus-engine/tests/llm_bench.cpp) registered as a target in the existing host and Android test CMake trees (which link thecactus_enginetarget directly — no hand-maintained archive lists).cactus testpath: download a prebuilt bundle, else convert + transpile. A local bundle directory is also accepted.docs/benchmarking.md: device preparation, core configuration, chunk-padding interpretation of prefill numbers, and bundle-vintage checks (row-major LM heads, pre-chunked-prefill text bundles).Verified
† measured on battery (power-limited; treat as a floor). Token-mode numbers match the previous standalone harness on the same device within noise, and the prefill ~204 tps / decode ~25.7 tps is consistent with the documented Samsung reference band (22–24 / 200+, verified 2026-06-09 on power).