Integrate changes made to stellar-rpc-blaster - #933
Conversation
| // fetchBlaster clones and builds stellar-rpc-blaster at the pinned commit. | ||
| func fetchBlaster(ctx context.Context, dir, repo string) (string, string, error) { | ||
| logger.Infof("fetching stellar-rpc-blaster (%s@dev)", repo) | ||
| logger.Infof("fetching stellar-rpc-blaster (%s@%s)", repo, blasterPin) |
There was a problem hiding this comment.
Flagging this, revert to @dev (or @main if we merge main<-dev) before merging (after commit merged to blaster of course)
| return "", "", fmt.Errorf("resolving blaster commit: %w", err) | ||
| for _, args := range [][]string{ | ||
| {"init", "-q", dir}, | ||
| {"-C", dir, "fetch", "-q", "--depth", "1", "https://github.com/" + repo + ".git", blasterPin}, |
There was a problem hiding this comment.
as above, revert to simpler git clone after since we'll no longer need to get blaster by SHA
🧪 Performance Evaluation Test #2Commit: ✅ Apply-load ingestion — verdict: ok📈 Ingest load test —
|
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1196.937 | 1114.485 / 1625.945 / 1912.928 | 3369.828 |
| load-test-ledgers-v27-sac | 1000 | 1097.646 | 1107.818 / 1179.879 / 1234.035 | 1303.658 |
| load-test-ledgers-v27-soroswap | 1000 | 791.306 | 802.133 / 867.097 / 924.901 | 1039.726 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3189.443s |
| Ingest busy-time | 3085.889s (96.8% utilization) |
| Per-ledger p50 / p95 / p99 | 1050.971 / 1414.227 / 1729.423 ms |
| Golden DB fetch+decompress | 2432s |
| stellar-core | v27.0.0 |
| Workflow run | #32781110895-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — 47bc5235013c
| Metric | Value |
|---|---|
| Ledgers ingested | 120960 ([63989120 -> 64110079]) |
| Retention window | 120960 |
| Wall-clock (total) | 3h5m15s |
| Ingest phase | 2h35m11s |
| Bulk-load finalize phase | 30m4s |
| Ledgers/sec (ingest) | 13.0 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — 47bc5235013c
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster dd8f7c36c0f5) against the backfilled RPC (ledgers [63989826, 64110785], handoff wait 1548s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents | 25 | 2995 | 15 (0.5%) | 4.6 | 64.2 | 2799.6 | 10010.6 |
| getHealth | 24 | 2820 | 0 (0.0%) | 1.9 | 11.2 | 106.6 | 186.1 |
| getLatestLedger | 19 | 2225 | 0 (0.0%) | 104.8 | 691.2 | 1009.2 | 1265.7 |
| getLedgers | 2 | 222 | 1 (0.5%) | 2588.7 | 9920.5 | 12730.4 | 15007.7 |
| getNetwork | 2 | 235 | 0 (0.0%) | 25.4 | 35.0 | 62.0 | 120.9 |
| getTransaction | 24 | 2819 | 0 (0.0%) | 31.7 | 52.2 | 74.9 | 199.8 |
| getTransactions | 4 | 459 | 0 (0.0%) | 135.9 | 382.7 | 448.5 | 643.6 |
getEvents results extended
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents/catch-up | 1 | 130 | 0 (0.0%) | 11.2 | 41.6 | 122.0 | 170.0 |
| getEvents/deep-pager | 5.75 | 695 | 0 (0.0%) | 3.3 | 57.8 | 184.2 | 641.5 |
| getEvents/deep-scan | 0.75 | 85 | 0 (0.0%) | 1.9 | 40.8 | 62.5 | 64.2 |
| getEvents/firehose | 0.5 | 59 | 0 (0.0%) | 3.9 | 29.1 | 46.6 | 71.7 |
| getEvents/head-poll | 12 | 1402 | 0 (0.0%) | 3.5 | 15.8 | 54.4 | 153.2 |
| getEvents/tail-poll | 3 | 372 | 15 (4.0%) | 8.1 | 4730.9 | 10010.6 | 10035.2 |
| getEvents/transfer-watcher | 2 | 252 | 0 (0.0%) | 8.1 | 18.9 | 21.5 | 54.7 |
✅ Go endpoint benchmarks — verdict: ok
Baseline v28.0.0 (e29c911f2a72) vs candidate 47bc5235013c — -benchmem -count=10, both refs sequentially on one box.
benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetLedgerRange-8 981.9n ± 1% 953.1n ± 2% -2.94% (p=0.001 n=10)
BatchGetLedgers-8 758.8µ ± 1% 794.2µ ± 2% +4.67% (p=0.000 n=10)
geomean 27.30µ 27.51µ +0.79%
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetLedgerRange-8 16.00 ± 0% 16.00 ± 0% ~ (p=1.000 n=10) ¹
BatchGetLedgers-8 474.7Ki ± 0% 520.7Ki ± 0% +9.71% (p=0.000 n=10)
geomean 2.723Ki 2.852Ki +4.74%
¹ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetLedgerRange-8 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) ¹
BatchGetLedgers-8 2.914k ± 0% 2.314k ± 0% -20.59% (p=0.000 n=10)
geomean 108.0 96.21 -10.89%
¹ all samples are equal
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 8.758µ ± 0% 8.790µ ± 0% +0.37% (p=0.000 n=10)
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 210.0µ ± 3% 213.2µ ± 2% +1.52% (p=0.023 n=10)
geomean 42.89µ 43.29µ +0.94%
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0% 565.4Ki ± 0% ~ (p=1.000 n=10)
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0% 30.00 ± 0% ~ (p=1.000 n=10) ¹
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetEventsTopicFilters-8 3.278m ± 5% 3.099m ± 10% ~ (p=0.353 n=10)
GetEvents-8 159.5µ ± ∞ ¹ 158.5µ ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.337m ± ∞ ¹ 1.329m ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 8.995µ ± 3% 8.819µ ± 2% ~ (p=0.190 n=10)
JSONTransactions/XDR_format-8 2.832µ ± 1% 2.784µ ± 1% -1.70% (p=0.000 n=10)
GetProtocolVersion-8 44.70µ ± ∞ ¹ 44.92µ ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 96.27µ 94.67µ -1.66%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetEventsTopicFilters-8 193.8Ki ± 4% 194.7Ki ± 2% ~ (p=0.853 n=10)
GetEvents-8 43.15Ki ± ∞ ¹ 43.15Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.324Mi ± ∞ ¹ 1.369Mi ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 2.000Ki ± 0% 2.000Ki ± 0% ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8 1.336Ki ± 0% 1.336Ki ± 0% ~ (p=1.000 n=10) ³
GetProtocolVersion-8 7.454Ki ± ∞ ¹ 7.907Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 24.68Ki 25.08Ki +1.62%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetEventsTopicFilters-8 1.550k ± 4% 1.558k ± 3% ~ (p=0.870 n=10)
GetEvents-8 621.0 ± ∞ ¹ 621.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 5.323k ± ∞ ¹ 4.722k ± ∞ ¹ ~ (p=1.000 n=1) ³
JSONTransactions/JSON_format-8 25.00 ± 0% 25.00 ± 0% ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8 23.00 ± 0% 23.00 ± 0% ~ (p=1.000 n=10) ²
GetProtocolVersion-8 120.0 ± ∞ ¹ 122.0 ± ∞ ¹ ~ (p=1.000 n=1) ³
geomean 265.9 261.6 -1.62%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
³ need >= 4 samples to detect a difference at alpha level 0.05
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetPreflight-8 824.0µ ± 1% 826.3µ ± 1% ~ (p=1.000 n=10)
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetPreflight-8 52.10Ki ± 0% 52.10Ki ± 0% ~ (p=0.337 n=10)
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetPreflight-8 290.0 ± 0% 290.0 ± 0% ~ (p=1.000 n=10)
Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/32781110895/go-bench/): baseline.txt, benchstat.txt, candidate.txt
Performance Evaluation Test #1
🧪 Performance Evaluation Test #1
Commit: c3b7060caf65 (use-blaster-update)
Run: https://github.com/stellar/stellar-rpc/actions/runs/32175408882
✅ Apply-load ingestion — verdict: ok
📈 Ingest load test — c3b7060
| Profile | Ledgers | ms/ledger | p50 / p95 / p99 ms | max ms |
|---|---|---|---|---|
| load-test-ledgers-v27-oz | 1000 | 1197.035 | 1115.607 / 1625.551 / 1910.703 | 2663.219 |
| load-test-ledgers-v27-sac | 1000 | 1098.067 | 1107.777 / 1181.001 / 1235.354 | 1302.337 |
| load-test-ledgers-v27-soroswap | 1000 | 791.126 | 802.408 / 866.016 / 922.562 | 1032.353 |
| Metric | Value |
|---|---|
| Ledgers replayed | 3000 |
| Initial DB ledger count | 120960 |
| Throughput | 0.94 ledgers/sec |
| Elapsed wall-clock | 3190.914s |
| Ingest busy-time | 3086.228s (96.7% utilization) |
| Per-ledger p50 / p95 / p99 | 1051.358 / 1420.913 / 1730.652 ms |
| Golden DB fetch+decompress | 2352s |
| stellar-core | v27.0.0 |
| Workflow run | #32175408882-1 |
✅ Backfill ingestion — verdict: ok
⏳ Backfill ingestion — c3b7060caf65
| Metric | Value |
|---|---|
| Ledgers ingested | 120960 ([63895872 -> 64016831]) |
| Retention window | 120960 |
| Wall-clock (total) | 3h12m52s |
| Ingest phase | 2h40m18s |
| Bulk-load finalize phase | 32m34s |
| Ledgers/sec (ingest) | 12.6 |
✅ Endpoint load test — verdict: ok
🎯 Endpoint load test — c3b7060caf65
Serial blast per endpoint (ramp-up 2m, duration 3m, blaster dd8f7c36c0f5) against the backfilled RPC (ledgers [63896573, 64017532], handoff wait 1503s).
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents | 25 | 2997 | 34 (1.1%) | 8.0 | 5365.8 | 10002.4 | 11960.3 |
| getHealth | 24 | 2819 | 0 (0.0%) | 0.6 | 14.6 | 140.3 | 243.2 |
| getLatestLedger | 19 | 2147 | 0 (0.0%) | 130.4 | 3975.2 | 5111.8 | 5586.9 |
| getLedgers | 2 | 148 | 3 (2.0%) | 2930.7 | 11362.3 | 15007.7 | 15007.7 |
| getNetwork | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getTransaction | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
| getTransactions | 0 | 0 | 0 (0.0%) | 0.0 | 0.0 | 0.0 | 0.0 |
getEvents results extended
| Endpoint | Target RPS | Requests | Errors | p50 (ms) | p95 (ms) | p99 (ms) | p99.9 (ms) |
|---|---|---|---|---|---|---|---|
| getEvents/catch-up | 1 | 114 | 0 (0.0%) | 27.0 | 2553.9 | 6799.4 | 7012.4 |
| getEvents/deep-pager | 5.75 | 669 | 0 (0.0%) | 4.0 | 3418.1 | 7110.7 | 8052.7 |
| getEvents/deep-scan | 0.75 | 95 | 0 (0.0%) | 5.6 | 2748.4 | 3493.9 | 6123.5 |
| getEvents/firehose | 0.5 | 62 | 0 (0.0%) | 5.4 | 2023.4 | 3901.4 | 7389.2 |
| getEvents/head-poll | 12 | 1452 | 0 (0.0%) | 5.5 | 3663.9 | 7303.2 | 8224.8 |
| getEvents/tail-poll | 3 | 347 | 34 (9.8%) | 23.8 | 10010.6 | 11960.3 | 15007.7 |
| getEvents/transfer-watcher | 2 | 258 | 0 (0.0%) | 13.7 | 5042.2 | 7114.8 | 8233.0 |
✅ Go endpoint benchmarks — verdict: ok
Baseline v28.0.0 (e29c911f2a72) vs candidate c3b7060caf65 — -benchmem -count=10, both refs sequentially on one box.
benchstat: baseline vs candidate
goos: linux
goarch: amd64
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/db
cpu: Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetLedgerRange-8 959.5n ± 1% 971.2n ± 1% +1.21% (p=0.002 n=10)
BatchGetLedgers-8 752.9µ ± 1% 762.1µ ± 3% ~ (p=0.853 n=10)
geomean 26.88µ 27.21µ +1.22%
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetLedgerRange-8 16.00 ± 0% 16.00 ± 0% ~ (p=1.000 n=10) ¹
BatchGetLedgers-8 474.7Ki ± 0% 474.7Ki ± 0% ~ (p=0.898 n=10)
geomean 2.723Ki 2.723Ki +0.00%
¹ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetLedgerRange-8 4.000 ± 0% 4.000 ± 0% ~ (p=1.000 n=10) ¹
BatchGetLedgers-8 2.914k ± 0% 2.914k ± 0% ~ (p=1.000 n=10) ¹
geomean 108.0 108.0 +0.00%
¹ all samples are equal
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/feewindow
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 8.738µ ± 0% 8.636µ ± 1% -1.16% (p=0.000 n=10)
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 223.7µ ± 2% 220.0µ ± 1% -1.66% (p=0.002 n=10)
geomean 44.21µ 43.59µ -1.41%
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 565.4Ki ± 0% 565.4Ki ± 0% ~ (p=0.474 n=10)
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
ComputeFeeDistribution/computeFeeDistribution-8 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹
ComputeFeeDistribution/alternativeComputeFeeDistribution-8 30.00 ± 0% 30.00 ± 0% ~ (p=1.000 n=10) ¹
geomean ² +0.00% ²
¹ all samples are equal
² summaries must be >0 to compute geomean
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/methods
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetEventsTopicFilters-8 2.618m ± 1% 2.599m ± 1% ~ (p=0.143 n=10)
GetEvents-8 165.8µ ± ∞ ¹ 157.8µ ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.340m ± ∞ ¹ 1.343m ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 8.561µ ± 1% 8.540µ ± 0% ~ (p=0.403 n=10)
JSONTransactions/XDR_format-8 2.803µ ± 0% 2.785µ ± 1% -0.62% (p=0.006 n=10)
GetProtocolVersion-8 43.04µ ± ∞ ¹ 44.62µ ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 91.85µ 91.43µ -0.45%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetEventsTopicFilters-8 190.8Ki ± 0% 190.6Ki ± 1% ~ (p=0.280 n=10)
GetEvents-8 43.15Ki ± ∞ ¹ 43.15Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 1.324Mi ± ∞ ¹ 1.324Mi ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 2.000Ki ± 0% 2.000Ki ± 0% ~ (p=1.000 n=10) ³
JSONTransactions/XDR_format-8 1.336Ki ± 0% 1.336Ki ± 0% ~ (p=1.000 n=10) ³
GetProtocolVersion-8 7.453Ki ± ∞ ¹ 7.455Ki ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 24.61Ki 24.61Ki -0.02%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetEventsTopicFilters-8 1.524k ± 0% 1.522k ± 1% ~ (p=0.338 n=10)
GetEvents-8 621.0 ± ∞ ¹ 621.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
GetLedgers-8 5.322k ± ∞ ¹ 5.322k ± ∞ ¹ ~ (p=1.000 n=1) ²
JSONTransactions/JSON_format-8 25.00 ± 0% 25.00 ± 0% ~ (p=1.000 n=10) ²
JSONTransactions/XDR_format-8 23.00 ± 0% 23.00 ± 0% ~ (p=1.000 n=10) ²
GetProtocolVersion-8 120.0 ± ∞ ¹ 120.0 ± ∞ ¹ ~ (p=1.000 n=1) ²
geomean 265.2 265.1 -0.03%
¹ need >= 6 samples for confidence interval at level 0.95
² all samples are equal
pkg: github.com/stellar/stellar-rpc/cmd/stellar-rpc/internal/preflight
│ baseline.txt │ candidate.txt │
│ sec/op │ sec/op vs base │
GetPreflight-8 808.4µ ± 0% 813.9µ ± 1% +0.69% (p=0.004 n=10)
│ baseline.txt │ candidate.txt │
│ B/op │ B/op vs base │
GetPreflight-8 52.10Ki ± 0% 52.10Ki ± 0% ~ (p=0.168 n=10)
│ baseline.txt │ candidate.txt │
│ allocs/op │ allocs/op vs base │
GetPreflight-8 290.0 ± 0% 290.0 ± 0% ~ (p=1.000 n=10)
Raw benchmark logs (s3://stellar-rpc-ci-load-test/runs/32175408882/go-bench/): baseline.txt, benchstat.txt, candidate.txt
|
Two questions:
|
What
Integrates recent changes made to
stellar-rpc-blasterbased on Overcat's endpoint traffic data. These changes are aimed at making stellar-rpc-blaster more effective at representing real traffic (i.e. request bodies are now generated with probabilities based on real data rather than intuition about what production traffic is likely to look like). Among many things, this includes an overhaul to how getEvents traffic is generated; realistic RPS proportions among endpoints; and distributions around variable parameters (such as limit) fit to those seen in real traffic.Please see this PR for the complete changes made.
Why
Recently, a release evaluation system was added to Stellar RPC. Among four tests included in the suite is one that evaluates RPC's performance serving queries. The requests it serves, as well as the report that it receives, are generated in
stellar-rpc-blaster. This PR and the work inblasteraims to make this test far better at representing/discovering regressions between releases that would happen in a production setting.Known Limitations
N/A