bench(bench): add real-world encoding/decoding benchmarks by sasa-tomic · Pull Request #723 · dfinity/candid

sasa-tomic · 2026-03-18T12:23:10Z

Motivation

Existing benchmarks miss critical decode paths; issue perf: Subtype checking should not perform quadratic traversal #603 notes zero coverage of service subtype checking.

Solution

Add 8 benchmarks covering production patterns: fully-populated NNS neurons, schema evolution (field skipping), vec of services (exercises check_subtype), Ok/Err result variants, wide records (34+ fields), large enums (21 arms), double optionals, and multi-arg encoding/decoding.

Details

Enable value feature on candid dependency for IDLValue/IDLArgs APIs.
Each benchmark reports separate encode/decode timings via bench_scope.

Relates to #603

### Motivation - Existing benchmarks miss critical decode paths; issue #603 notes zero coverage of service subtype checking. ### Solution - Add 8 benchmarks covering production patterns: fully-populated NNS neurons, schema evolution (field skipping), vec of services (exercises check_subtype), Ok/Err result variants, wide records (34+ fields), large enums (21 arms), double optionals, and multi-arg encoding/decoding. ### Details - Enable `value` feature on candid dependency for IDLValue/IDLArgs APIs. - Each benchmark reports separate encode/decode timings via bench_scope. Relates to #603

github-actions · 2026-03-18T12:37:57Z

Name	Max Mem (Kb)	Encode	Decode
blob	4_224	4_207_487	2_122_059
btreemap	75_456	536_170_085 ($\textcolor{red}{0.79\%}$)	10_097_213_269 ($\textcolor{green}{-0.87\%}$)
double_option (new)	128	1_317_074	28_888_466
large_variant (new)	320	1_043_241	21_118_880
multi_arg (new)	64	549_516	6_560_643
nns	192	2_021_253	5_630_551 ($\textcolor{green}{-0.09\%}$)
nns_list_neurons (new)	1_152	6_638_914	220_220_685
nns_list_proposal	1_216	7_022_215 ($\textcolor{red}{0.38\%}$)	60_879_051 ($\textcolor{green}{-1.57\%}$)
option_list	128	716_687 ($\textcolor{red}{0.03\%}$)	16_877_235 ($\textcolor{green}{-0.52\%}$)
result_variant (new)	192	1_322_866	17_911_344
subtype_decode (new)	512	2_652_101	50_699_398
text	6_336	4_204_384	7_877_508
variant_list	128	711_131 ($\textcolor{green}{-0.01\%}$)	15_903_518 ($\textcolor{green}{-1.52\%}$)
vec_int16	12_480	8_404_689	249_586_272
vec_nat	11_008	67_095_666	277_077_493 ($\textcolor{green}{-0.19\%}$)
vec_nat32	24_768	16_793_297	243_295_105
vec_nat64	49_344	33_570_495	251_683_977
vec_service (new)	64	781_165	96_311_262
wide_record (new)	1_152	3_268_654	48_410_039

Parser cost: 16_174_059 ($\textcolor{green}{-0.00\%}$)
Extra args: 2_859_770

Click to see raw report

---------------------------------------------------

Benchmark: blob
  total:
    instructions: 6.33 M (0.00%) (change within noise threshold)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.21 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 2.12 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: btreemap
  total:
    instructions: 10.63 B (-0.79%) (change within noise threshold)
    heap_increase: 1179 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 536.17 M (0.79%) (change within noise threshold)
    heap_increase: 159 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 10.10 B (-0.87%) (change within noise threshold)
    heap_increase: 1020 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: double_option (new)
  total:
    instructions: 30.21 M (new)
    heap_increase: 2 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 1.32 M (new)
    heap_increase: 0 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 28.89 M (new)
    heap_increase: 2 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: extra_args
  total:
    instructions: 2.86 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: large_variant (new)
  total:
    instructions: 22.16 M (new)
    heap_increase: 5 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 1.04 M (new)
    heap_increase: 2 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 21.12 M (new)
    heap_increase: 3 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: multi_arg (new)
  total:
    instructions: 7.11 M (new)
    heap_increase: 1 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 549.52 K (new)
    heap_increase: 1 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 6.56 M (new)
    heap_increase: 0 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: nns
  total:
    instructions: 24.66 M (-0.02%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  0. Parsing (scope):
    calls: 1 (no change)
    instructions: 16.17 M (-0.00%) (change within noise threshold)
    heap_increase: 3 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 2.02 M (no change)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 5.63 M (-0.09%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: nns_list_neurons (new)
  total:
    instructions: 226.86 M (new)
    heap_increase: 18 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 6.64 M (new)
    heap_increase: 18 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 220.22 M (new)
    heap_increase: 0 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: nns_list_proposal
  total:
    instructions: 67.90 M (-1.37%) (change within noise threshold)
    heap_increase: 19 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 7.02 M (0.38%) (change within noise threshold)
    heap_increase: 5 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 60.88 M (-1.57%) (change within noise threshold)
    heap_increase: 14 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: option_list
  total:
    instructions: 17.60 M (-0.50%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 716.69 K (0.03%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 16.88 M (-0.52%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: result_variant (new)
  total:
    instructions: 19.24 M (new)
    heap_increase: 3 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 1.32 M (new)
    heap_increase: 1 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 17.91 M (new)
    heap_increase: 2 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: subtype_decode (new)
  total:
    instructions: 53.35 M (new)
    heap_increase: 8 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 2.65 M (new)
    heap_increase: 8 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 50.70 M (new)
    heap_increase: 0 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: text
  total:
    instructions: 12.08 M (no change)
    heap_increase: 99 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 4.20 M (no change)
    heap_increase: 66 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 7.88 M (no change)
    heap_increase: 33 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: variant_list
  total:
    instructions: 16.62 M (-1.46%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 711.13 K (-0.01%) (change within noise threshold)
    heap_increase: 0 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 15.90 M (-1.52%) (change within noise threshold)
    heap_increase: 2 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_int16
  total:
    instructions: 257.99 M (no change)
    heap_increase: 195 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 8.40 M (no change)
    heap_increase: 130 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 249.59 M (no change)
    heap_increase: 65 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_nat
  total:
    instructions: 344.18 M (-0.15%) (change within noise threshold)
    heap_increase: 172 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 67.10 M (no change)
    heap_increase: 33 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 277.08 M (-0.19%) (change within noise threshold)
    heap_increase: 139 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_nat32
  total:
    instructions: 260.09 M (no change)
    heap_increase: 387 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 16.79 M (no change)
    heap_increase: 258 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 243.30 M (no change)
    heap_increase: 129 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_nat64
  total:
    instructions: 285.26 M (no change)
    heap_increase: 771 pages (no change)
    stable_memory_increase: 0 pages (no change)

  1. Encoding (scope):
    calls: 1 (no change)
    instructions: 33.57 M (no change)
    heap_increase: 514 pages (no change)
    stable_memory_increase: 0 pages (no change)

  2. Decoding (scope):
    calls: 1 (no change)
    instructions: 251.68 M (no change)
    heap_increase: 257 pages (no change)
    stable_memory_increase: 0 pages (no change)

---------------------------------------------------

Benchmark: vec_service (new)
  total:
    instructions: 97.09 M (new)
    heap_increase: 1 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 781.16 K (new)
    heap_increase: 0 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 96.31 M (new)
    heap_increase: 1 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Benchmark: wide_record (new)
  total:
    instructions: 51.68 M (new)
    heap_increase: 18 pages (new)
    stable_memory_increase: 0 pages (new)

  1. Encoding (scope):
    calls: 1 (new)
    instructions: 3.27 M (new)
    heap_increase: 18 pages (new)
    stable_memory_increase: 0 pages (new)

  2. Decoding (scope):
    calls: 1 (new)
    instructions: 48.41 M (new)
    heap_increase: 0 pages (new)
    stable_memory_increase: 0 pages (new)

---------------------------------------------------

Summary:
  instructions:
    status:   New benchmarks added ➕
    counts:   [total 20 | regressed 0 | improved 0 | new 8 | unchanged 12]
    change:   [max +16 | p75 0 | median -2.79K | p25 -315.54K | min -84.49M]
    change %: [max 0.00% | p75 0.00% | median -0.01% | p25 -0.57% | min -1.46%]

  heap_increase:
    status:   New benchmarks added ➕
    counts:   [total 20 | regressed 0 | improved 0 | new 8 | unchanged 12]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

  stable_memory_increase:
    status:   New benchmarks added ➕
    counts:   [total 20 | regressed 0 | improved 0 | new 8 | unchanged 12]
    change:   [max 0 | p75 0 | median 0 | p25 0 | min 0]
    change %: [max 0.00% | p75 0.00% | median 0.00% | p25 0.00% | min 0.00%]

---------------------------------------------------

Only significant changes:
| status | name                          | calls |     ins |  ins Δ% | HI |  HI Δ% | SMI |  SMI Δ% |
|--------|-------------------------------|-------|---------|---------|----|--------|-----|---------|
|  new   | double_option                 |       |  30.21M |         |  2 |        |   0 |         |
|  new   | double_option::1. Encoding    |     1 |   1.32M |         |  0 |        |   0 |         |
|  new   | double_option::2. Decoding    |     1 |  28.89M |         |  2 |        |   0 |         |
|  new   | large_variant                 |       |  22.16M |         |  5 |        |   0 |         |
|  new   | large_variant::1. Encoding    |     1 |   1.04M |         |  2 |        |   0 |         |
|  new   | large_variant::2. Decoding    |     1 |  21.12M |         |  3 |        |   0 |         |
|  new   | multi_arg                     |       |   7.11M |         |  1 |        |   0 |         |
|  new   | multi_arg::1. Encoding        |     1 | 549.52K |         |  1 |        |   0 |         |
|  new   | multi_arg::2. Decoding        |     1 |   6.56M |         |  0 |        |   0 |         |
|  new   | nns_list_neurons              |       | 226.86M |         | 18 |        |   0 |         |
|  new   | nns_list_neurons::1. Encoding |     1 |   6.64M |         | 18 |        |   0 |         |
|  new   | nns_list_neurons::2. Decoding |     1 | 220.22M |         |  0 |        |   0 |         |
|  new   | result_variant                |       |  19.24M |         |  3 |        |   0 |         |
|  new   | result_variant::1. Encoding   |     1 |   1.32M |         |  1 |        |   0 |         |
|  new   | result_variant::2. Decoding   |     1 |  17.91M |         |  2 |        |   0 |         |
|  new   | subtype_decode                |       |  53.35M |         |  8 |        |   0 |         |
|  new   | subtype_decode::1. Encoding   |     1 |   2.65M |         |  8 |        |   0 |         |
|  new   | subtype_decode::2. Decoding   |     1 |  50.70M |         |  0 |        |   0 |         |
|  new   | vec_service                   |       |  97.09M |         |  1 |        |   0 |         |
|  new   | vec_service::1. Encoding      |     1 | 781.16K |         |  0 |        |   0 |         |
|  new   | vec_service::2. Decoding      |     1 |  96.31M |         |  1 |        |   0 |         |
|  new   | wide_record                   |       |  51.68M |         | 18 |        |   0 |         |
|  new   | wide_record::1. Encoding      |     1 |   3.27M |         | 18 |        |   0 |         |
|  new   | wide_record::2. Decoding      |     1 |  48.41M |         |  0 |        |   0 |         |

ins = instructions, HI = heap_increase, SMI = stable_memory_increase, Δ% = percent change

---------------------------------------------------
Successfully persisted results to canbench_results.yml

sasa-tomic self-assigned this Mar 18, 2026

sasa-tomic marked this pull request as ready for review March 18, 2026 12:23

sasa-tomic requested a review from a team as a code owner March 18, 2026 12:23

sasa-tomic mentioned this pull request Mar 18, 2026

perf: Subtype checking should not perform quadratic traversal #603

Open

fix naming

265d52b

Merge branch 'master' into bench/add-real-world-patterns

031f082

lwshang approved these changes Mar 18, 2026

View reviewed changes

lwshang merged commit 6b0576e into master Mar 18, 2026
11 checks passed

lwshang deleted the bench/add-real-world-patterns branch March 18, 2026 16:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bench(bench): add real-world encoding/decoding benchmarks#723

bench(bench): add real-world encoding/decoding benchmarks#723
lwshang merged 3 commits intomasterfrom
bench/add-real-world-patterns

sasa-tomic commented Mar 18, 2026

Uh oh!

github-actions bot commented Mar 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

sasa-tomic commented Mar 18, 2026

Motivation

Solution

Details

Uh oh!

github-actions bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions bot commented Mar 18, 2026 •

edited

Loading