Skip to content

Integrate changes made to stellar-rpc-blaster - #933

Draft
cjonas9 wants to merge 4 commits into
mainfrom
use-blaster-update
Draft

Integrate changes made to stellar-rpc-blaster#933
cjonas9 wants to merge 4 commits into
mainfrom
use-blaster-update

Conversation

@cjonas9

@cjonas9 cjonas9 commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

What

Integrates recent changes made to stellar-rpc-blaster based on Overcat's endpoint traffic data. These changes are aimed at making stellar-rpc-blaster more effective at representing real traffic (i.e. request bodies are now generated with probabilities based on real data rather than intuition about what production traffic is likely to look like). Among many things, this includes an overhaul to how getEvents traffic is generated; realistic RPS proportions among endpoints; and distributions around variable parameters (such as limit) fit to those seen in real traffic.

Please see this PR for the complete changes made.

Why

Recently, a release evaluation system was added to Stellar RPC. Among four tests included in the suite is one that evaluates RPC's performance serving queries. The requests it serves, as well as the report that it receives, are generated in stellar-rpc-blaster. This PR and the work in blaster aims to make this test far better at representing/discovering regressions between releases that would happen in a production setting.

Known Limitations

N/A

// fetchBlaster clones and builds stellar-rpc-blaster at the pinned commit.
func fetchBlaster(ctx context.Context, dir, repo string) (string, string, error) {
logger.Infof("fetching stellar-rpc-blaster (%s@dev)", repo)
logger.Infof("fetching stellar-rpc-blaster (%s@%s)", repo, blasterPin)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Flagging this, revert to @dev (or @main if we merge main<-dev) before merging (after commit merged to blaster of course)

return "", "", fmt.Errorf("resolving blaster commit: %w", err)
for _, args := range [][]string{
{"init", "-q", dir},
{"-C", dir, "fetch", "-q", "--depth", "1", "https://github.com/" + repo + ".git", blasterPin},

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as above, revert to simpler git clone after since we'll no longer need to get blaster by SHA

@github-actions

github-actions Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

🧪 Performance Evaluation Test #2

Commit: 47bc5235013c (use-blaster-update)
Run: https://github.com/stellar/stellar-rpc/actions/runs/32781110895

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — 47bc523

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1196.937 1114.485 / 1625.945 / 1912.928 3369.828
load-test-ledgers-v27-sac 1000 1097.646 1107.818 / 1179.879 / 1234.035 1303.658
load-test-ledgers-v27-soroswap 1000 791.306 802.133 / 867.097 / 924.901 1039.726
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3189.443s
Ingest busy-time 3085.889s (96.8% utilization)
Per-ledger p50 / p95 / p99 1050.971 / 1414.227 / 1729.423 ms
Golden DB fetch+decompress 2432s
stellar-core v27.0.0
Workflow run #32781110895-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — 47bc5235013c

Metric Value
Ledgers ingested 120960 ([63989120 -> 64110079])
Retention window 120960
Wall-clock (total) 3h5m15s
Ingest phase 2h35m11s
Bulk-load finalize phase 30m4s
Ledgers/sec (ingest) 13.0

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — 47bc5235013c

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster dd8f7c36c0f5) against the backfilled RPC (ledgers [63989826, 64110785], handoff wait 1548s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getEvents 25 2995 15 (0.5%) 4.6 64.2 2799.6 10010.6
getHealth 24 2820 0 (0.0%) 1.9 11.2 106.6 186.1
getLatestLedger 19 2225 0 (0.0%) 104.8 691.2 1009.2 1265.7
getLedgers 2 222 1 (0.5%) 2588.7 9920.5 12730.4 15007.7
getNetwork 2 235 0 (0.0%) 25.4 35.0 62.0 120.9
getTransaction 24 2819 0 (0.0%) 31.7 52.2 74.9 199.8
getTransactions 4 459 0 (0.0%) 135.9 382.7 448.5 643.6
getEvents results extended
Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getEvents/catch-up 1 130 0 (0.0%) 11.2 41.6 122.0 170.0
getEvents/deep-pager 5.75 695 0 (0.0%) 3.3 57.8 184.2 641.5
getEvents/deep-scan 0.75 85 0 (0.0%) 1.9 40.8 62.5 64.2
getEvents/firehose 0.5 59 0 (0.0%) 3.9 29.1 46.6 71.7
getEvents/head-poll 12 1402 0 (0.0%) 3.5 15.8 54.4 153.2
getEvents/tail-poll 3 372 15 (4.0%) 8.1 4730.9 10010.6 10035.2
getEvents/transfer-watcher 2 252 0 (0.0%) 8.1 18.9 21.5 54.7

✅ Go endpoint benchmarks — verdict: ok

Baseline v28.0.0 (e29c911f2a72) vs candidate 47bc5235013c-benchmem -count=10, both refs sequentially on one box.

benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
                  │ baseline.txt │           candidate.txt            │
                  │    sec/op    │   sec/op     vs base               │
GetLedgerRange-8     981.9n ± 1%   953.1n ± 2%  -2.94% (p=0.001 n=10)
BatchGetLedgers-8    758.8µ ± 1%   794.2µ ± 2%  +4.67% (p=0.000 n=10)
geomean              27.30µ        27.51µ       +0.79%

                  │ baseline.txt │             candidate.txt             │
                  │     B/op     │     B/op      vs base                 │
GetLedgerRange-8      16.00 ± 0%     16.00 ± 0%       ~ (p=1.000 n=10) ¹
BatchGetLedgers-8   474.7Ki ± 0%   520.7Ki ± 0%  +9.71% (p=0.000 n=10)
geomean             2.723Ki        2.852Ki       +4.74%
¹ all samples are equal

                  │ baseline.txt │             candidate.txt             │
                  │  allocs/op   │  allocs/op   vs base                  │
GetLedgerRange-8      4.000 ± 0%    4.000 ± 0%        ~ (p=1.000 n=10) ¹
BatchGetLedgers-8    2.914k ± 0%   2.314k ± 0%  -20.59% (p=0.000 n=10)
geomean               108.0         96.21       -10.89%
¹ all samples are equal

pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
                                                           │ baseline.txt │           candidate.txt            │
                                                           │    sec/op    │   sec/op     vs base               │
ComputeFeeDistribution/computeFeeDistribution-8               8.758µ ± 0%   8.790µ ± 0%  +0.37% (p=0.000 n=10)
ComputeFeeDistribution/alternativeComputeFeeDistribution-8    210.0µ ± 3%   213.2µ ± 2%  +1.52% (p=0.023 n=10)
geomean                                                       42.89µ        43.29µ       +0.94%

                                                           │  baseline.txt  │             candidate.txt             │
                                                           │      B/op      │     B/op      vs base                 │
ComputeFeeDistribution/computeFeeDistribution-8                0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8   565.4Ki ± 0%     565.4Ki ± 0%       ~ (p=1.000 n=10)
geomean                                                                   ²                 +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                           │ baseline.txt │            candidate.txt            │
                                                           │  allocs/op   │ allocs/op   vs base                 │
ComputeFeeDistribution/computeFeeDistribution-8              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8   30.00 ± 0%     30.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                                 ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
                               │ baseline.txt  │              candidate.txt              │
                               │    sec/op     │     sec/op      vs base                 │
GetEventsTopicFilters-8          3.278m ± 5%     3.099m ± 10%         ~ (p=0.353 n=10)
GetEvents-8                      159.5µ ±  ∞ ¹   158.5µ ±   ∞ ¹       ~ (p=1.000 n=1)  ²
GetLedgers-8                     1.337m ±  ∞ ¹   1.329m ±   ∞ ¹       ~ (p=1.000 n=1)  ²
JSONTransactions/JSON_format-8   8.995µ ± 3%     8.819µ ±  2%         ~ (p=0.190 n=10)
JSONTransactions/XDR_format-8    2.832µ ± 1%     2.784µ ±  1%    -1.70% (p=0.000 n=10)
GetProtocolVersion-8             44.70µ ±  ∞ ¹   44.92µ ±   ∞ ¹       ~ (p=1.000 n=1)  ²
geomean                          96.27µ          94.67µ          -1.66%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                               │  baseline.txt  │              candidate.txt              │
                               │      B/op      │      B/op       vs base                 │
GetEventsTopicFilters-8          193.8Ki ± 4%     194.7Ki ± 2%         ~ (p=0.853 n=10)
GetEvents-8                      43.15Ki ±  ∞ ¹   43.15Ki ±  ∞ ¹       ~ (p=1.000 n=1)  ²
GetLedgers-8                     1.324Mi ±  ∞ ¹   1.369Mi ±  ∞ ¹       ~ (p=1.000 n=1)  ²
JSONTransactions/JSON_format-8   2.000Ki ± 0%     2.000Ki ± 0%         ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8    1.336Ki ± 0%     1.336Ki ± 0%         ~ (p=1.000 n=10) ³
GetProtocolVersion-8             7.454Ki ±  ∞ ¹   7.907Ki ±  ∞ ¹       ~ (p=1.000 n=1)  ²
geomean                          24.68Ki          25.08Ki         +1.62%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal

                               │ baseline.txt  │             candidate.txt              │
                               │   allocs/op   │   allocs/op    vs base                 │
GetEventsTopicFilters-8          1.550k ± 4%     1.558k ± 3%         ~ (p=0.870 n=10)
GetEvents-8                       621.0 ±  ∞ ¹    621.0 ±  ∞ ¹       ~ (p=1.000 n=1)  ²
GetLedgers-8                     5.323k ±  ∞ ¹   4.722k ±  ∞ ¹       ~ (p=1.000 n=1)  ³
JSONTransactions/JSON_format-8    25.00 ± 0%      25.00 ± 0%         ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8     23.00 ± 0%      23.00 ± 0%         ~ (p=1.000 n=10) ²
GetProtocolVersion-8              120.0 ±  ∞ ¹    122.0 ±  ∞ ¹       ~ (p=1.000 n=1)  ³
geomean                           265.9           261.6         -1.62%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ need >= 4 samples to detect a difference at alpha level 0.05

pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
               │ baseline.txt │         candidate.txt         │
               │    sec/op    │   sec/op     vs base          │
GetPreflight-8    824.0µ ± 1%   826.3µ ± 1%  ~ (p=1.000 n=10)

               │ baseline.txt │         candidate.txt          │
               │     B/op     │     B/op      vs base          │
GetPreflight-8   52.10Ki ± 0%   52.10Ki ± 0%  ~ (p=0.337 n=10)

               │ baseline.txt │        candidate.txt         │
               │  allocs/op   │ allocs/op   vs base          │
GetPreflight-8     290.0 ± 0%   290.0 ± 0%  ~ (p=1.000 n=10)

Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/32781110895/go-bench/): baseline.txt, benchstat.txt, candidate.txt

Performance Evaluation Test #1

🧪 Performance Evaluation Test #1

Commit: c3b7060caf65 (use-blaster-update)
Run: https://github.com/stellar/stellar-rpc/actions/runs/32175408882

✅ Apply-load ingestion — verdict: ok

📈 Ingest load test — c3b7060

Profile Ledgers ms/ledger p50 / p95 / p99 ms max ms
load-test-ledgers-v27-oz 1000 1197.035 1115.607 / 1625.551 / 1910.703 2663.219
load-test-ledgers-v27-sac 1000 1098.067 1107.777 / 1181.001 / 1235.354 1302.337
load-test-ledgers-v27-soroswap 1000 791.126 802.408 / 866.016 / 922.562 1032.353
Metric Value
Ledgers replayed 3000
Initial DB ledger count 120960
Throughput 0.94 ledgers/sec
Elapsed wall-clock 3190.914s
Ingest busy-time 3086.228s (96.7% utilization)
Per-ledger p50 / p95 / p99 1051.358 / 1420.913 / 1730.652 ms
Golden DB fetch+decompress 2352s
stellar-core v27.0.0
Workflow run #32175408882-1

✅ Backfill ingestion — verdict: ok

⏳ Backfill ingestion — c3b7060caf65

Metric Value
Ledgers ingested 120960 ([63895872 -> 64016831])
Retention window 120960
Wall-clock (total) 3h12m52s
Ingest phase 2h40m18s
Bulk-load finalize phase 32m34s
Ledgers/sec (ingest) 12.6

✅ Endpoint load test — verdict: ok

🎯 Endpoint load test — c3b7060caf65

Serial blast per endpoint (ramp-up 2m, duration 3m, blaster dd8f7c36c0f5) against the backfilled RPC (ledgers [63896573, 64017532], handoff wait 1503s).

Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getEvents 25 2997 34 (1.1%) 8.0 5365.8 10002.4 11960.3
getHealth 24 2819 0 (0.0%) 0.6 14.6 140.3 243.2
getLatestLedger 19 2147 0 (0.0%) 130.4 3975.2 5111.8 5586.9
getLedgers 2 148 3 (2.0%) 2930.7 11362.3 15007.7 15007.7
getNetwork 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getTransaction 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getTransactions 0 0 0 (0.0%) 0.0 0.0 0.0 0.0
getEvents results extended
Endpoint Target RPS Requests Errors p50 (ms) p95 (ms) p99 (ms) p99.9 (ms)
getEvents/catch-up 1 114 0 (0.0%) 27.0 2553.9 6799.4 7012.4
getEvents/deep-pager 5.75 669 0 (0.0%) 4.0 3418.1 7110.7 8052.7
getEvents/deep-scan 0.75 95 0 (0.0%) 5.6 2748.4 3493.9 6123.5
getEvents/firehose 0.5 62 0 (0.0%) 5.4 2023.4 3901.4 7389.2
getEvents/head-poll 12 1452 0 (0.0%) 5.5 3663.9 7303.2 8224.8
getEvents/tail-poll 3 347 34 (9.8%) 23.8 10010.6 11960.3 15007.7
getEvents/transfer-watcher 2 258 0 (0.0%) 13.7 5042.2 7114.8 8233.0

✅ Go endpoint benchmarks — verdict: ok

Baseline v28.0.0 (e29c911f2a72) vs candidate c3b7060caf65-benchmem -count=10, both refs sequentially on one box.

benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
                  │ baseline.txt │           candidate.txt            │
                  │    sec/op    │   sec/op     vs base               │
GetLedgerRange-8     959.5n ± 1%   971.2n ± 1%  +1.21% (p=0.002 n=10)
BatchGetLedgers-8    752.9µ ± 1%   762.1µ ± 3%       ~ (p=0.853 n=10)
geomean              26.88µ        27.21µ       +1.22%

                  │ baseline.txt │             candidate.txt             │
                  │     B/op     │     B/op      vs base                 │
GetLedgerRange-8      16.00 ± 0%     16.00 ± 0%       ~ (p=1.000 n=10) ¹
BatchGetLedgers-8   474.7Ki ± 0%   474.7Ki ± 0%       ~ (p=0.898 n=10)
geomean             2.723Ki        2.723Ki       +0.00%
¹ all samples are equal

                  │ baseline.txt │            candidate.txt             │
                  │  allocs/op   │  allocs/op   vs base                 │
GetLedgerRange-8      4.000 ± 0%    4.000 ± 0%       ~ (p=1.000 n=10) ¹
BatchGetLedgers-8    2.914k ± 0%   2.914k ± 0%       ~ (p=1.000 n=10) ¹
geomean               108.0         108.0       +0.00%
¹ all samples are equal

pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
                                                           │ baseline.txt │           candidate.txt            │
                                                           │    sec/op    │   sec/op     vs base               │
ComputeFeeDistribution/computeFeeDistribution-8               8.738µ ± 0%   8.636µ ± 1%  -1.16% (p=0.000 n=10)
ComputeFeeDistribution/alternativeComputeFeeDistribution-8    223.7µ ± 2%   220.0µ ± 1%  -1.66% (p=0.002 n=10)
geomean                                                       44.21µ        43.59µ       -1.41%

                                                           │  baseline.txt  │             candidate.txt             │
                                                           │      B/op      │     B/op      vs base                 │
ComputeFeeDistribution/computeFeeDistribution-8                0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8   565.4Ki ± 0%     565.4Ki ± 0%       ~ (p=0.474 n=10)
geomean                                                                   ²                 +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                           │ baseline.txt │            candidate.txt            │
                                                           │  allocs/op   │ allocs/op   vs base                 │
ComputeFeeDistribution/computeFeeDistribution-8              0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8   30.00 ± 0%     30.00 ± 0%       ~ (p=1.000 n=10) ¹
geomean                                                                 ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
                               │ baseline.txt  │             candidate.txt              │
                               │    sec/op     │    sec/op      vs base                 │
GetEventsTopicFilters-8          2.618m ± 1%     2.599m ± 1%         ~ (p=0.143 n=10)
GetEvents-8                      165.8µ ±  ∞ ¹   157.8µ ±  ∞ ¹       ~ (p=1.000 n=1)  ²
GetLedgers-8                     1.340m ±  ∞ ¹   1.343m ±  ∞ ¹       ~ (p=1.000 n=1)  ²
JSONTransactions/JSON_format-8   8.561µ ± 1%     8.540µ ± 0%         ~ (p=0.403 n=10)
JSONTransactions/XDR_format-8    2.803µ ± 0%     2.785µ ± 1%    -0.62% (p=0.006 n=10)
GetProtocolVersion-8             43.04µ ±  ∞ ¹   44.62µ ±  ∞ ¹       ~ (p=1.000 n=1)  ²
geomean                          91.85µ          91.43µ         -0.45%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                               │  baseline.txt  │              candidate.txt              │
                               │      B/op      │      B/op       vs base                 │
GetEventsTopicFilters-8          190.8Ki ± 0%     190.6Ki ± 1%         ~ (p=0.280 n=10)
GetEvents-8                      43.15Ki ±  ∞ ¹   43.15Ki ±  ∞ ¹       ~ (p=1.000 n=1)  ²
GetLedgers-8                     1.324Mi ±  ∞ ¹   1.324Mi ±  ∞ ¹       ~ (p=1.000 n=1)  ²
JSONTransactions/JSON_format-8   2.000Ki ± 0%     2.000Ki ± 0%         ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8    1.336Ki ± 0%     1.336Ki ± 0%         ~ (p=1.000 n=10) ³
GetProtocolVersion-8             7.453Ki ±  ∞ ¹   7.455Ki ±  ∞ ¹       ~ (p=1.000 n=1)  ²
geomean                          24.61Ki          24.61Ki         -0.02%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal

                               │ baseline.txt  │             candidate.txt              │
                               │   allocs/op   │   allocs/op    vs base                 │
GetEventsTopicFilters-8          1.524k ± 0%     1.522k ± 1%         ~ (p=0.338 n=10)
GetEvents-8                       621.0 ±  ∞ ¹    621.0 ±  ∞ ¹       ~ (p=1.000 n=1)  ²
GetLedgers-8                     5.322k ±  ∞ ¹   5.322k ±  ∞ ¹       ~ (p=1.000 n=1)  ²
JSONTransactions/JSON_format-8    25.00 ± 0%      25.00 ± 0%         ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8     23.00 ± 0%      23.00 ± 0%         ~ (p=1.000 n=10) ²
GetProtocolVersion-8              120.0 ±  ∞ ¹    120.0 ±  ∞ ¹       ~ (p=1.000 n=1)  ²
geomean                           265.2           265.1         -0.03%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal

pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
               │ baseline.txt │           candidate.txt            │
               │    sec/op    │   sec/op     vs base               │
GetPreflight-8    808.4µ ± 0%   813.9µ ± 1%  +0.69% (p=0.004 n=10)

               │ baseline.txt │         candidate.txt          │
               │     B/op     │     B/op      vs base          │
GetPreflight-8   52.10Ki ± 0%   52.10Ki ± 0%  ~ (p=0.168 n=10)

               │ baseline.txt │        candidate.txt         │
               │  allocs/op   │ allocs/op   vs base          │
GetPreflight-8     290.0 ± 0%   290.0 ± 0%  ~ (p=1.000 n=10)

Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/32175408882/go-bench/): baseline.txt, benchstat.txt, candidate.txt

@roebee

roebee commented Aug 23, 2026

Copy link
Copy Markdown

Two questions:

  1. The body lists [TODO]: which specific blaster changes from the Overcat analysis must be integrated?
  2. What validates the integration: the endpoint load-test profile, or a full campaign run?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants