RPC v2 — Roadmap

[tamirms]

RPC v2 — Roadmap

Living roadmap for shipping Stellar RPC v2. At a minimum, it's refreshed each sprint to set the next sprint's targets.

Status: ✅ done · 🟡 in progress · ⬜ not started.
Priority: P0 must-ship for MVP · P1 important, not MVP-blocking · P2 deferrable · P3 furthest out.

🎯 Sprint 75 — target

• Query serving
  • implement the v2 events API; support the v1 events API in rpc v2
    • P0 — Implement the v2 events endpoint and deprecate the existing getEvents #774 — @urvisavla
  • serve the API from the v2 stores: handlers shared with v1, rpc v1 and v2 built as separate binaries from the same repo/branch
    • P0 — Wire the v2 stores into the API handlers and support both v1 and v2 #772 — @karthikiyer56
• Benchmarks
  • experiments and writeup to achieve the Phase 1–3 ingestion targets
    • P0 — Investigate and fix the issues surfaced by the O3 ingestion load test #800 — @tamirms
  • query benchmark commands: reads across all data types
    • P1 — Full-history query benchmark commands — reads across all data types  #856 — @marwen-abid
  • read-endpoint SLAs under ingestion load
    • P1 — Read-endpoint SLAs under ingestion load #870 — @marwen-abid
  • PoC: speed up transaction-status reads (ledger hint on getTransaction, WebSocket status push)
    • P2 — [PoC] speed up transaction-status reads: ledger hint on getTransaction, WebSocket status push #905 — @marwen-abid
  • PoC: remote captive Core service, streaming ledgers to RPC over the network
    • P2 — [PoC] Remote captive Core service: stream ledgers to RPC over the network #906 — @marwen-abid

---

The work to ship an MVP release of RPC v2 falls under the categories below. Each sprint, we make progress by completing tasks across any number of them.

Not all work is broken into issues yet — some is design-gated. We settle the design first, then create the implementation issues from it.

DB layer — ✅ · P0

When complete: any ledger, transaction, or event from anywhere in history can be written and read back — recent data in a live, writable database, older data in immutable files written once and never changed. These are the storage read/write primitives; wiring them to the JSON-RPC endpoints comes later (Query serving).

Data type	Status	Issues
Ledger	✅	#695 / #725 / #583 / #584
Events	✅	core #756 ✅, query engine #663 ✅
TxHash	✅	#729, #728
XDR view extractors	✅	#764
Ingestion	✅	#765

Unified ingestion workflow — ✅ · P0

When complete: one daemon mode that catches up on startup (backfilling history as a subroutine), then ingests live ledgers from captive core — freezing completed chunks from the live database into immutable files, and pruning data once it ages out of the retention window.

• Backfill (catch-up subroutine) — design ✅ (Checking in full-history design docs into stellar-rpc for review #617)
• Live ingestion + freeze/prune — design ✅ (Streaming workflow design (catch-up, live ingestion, freeze & prune) #722)
• Streaming daemon (implements both) — ✅ ([Epic] Unified ingestion workflow: full-history streaming daemon #808)

Query serving — 🟡 · P0

When complete: the service answers all read endpoints from the v2 stores — routing across hot + cold — through handlers shared with v1, which keeps its SQLite path through a deprecation window, with the events API on its v2 form.

• Query serving across hot + cold stores under concurrent ingestion/lifecycle — ✅
  • ✅ Design: concurrent query serving across all data types #770
  • ✅ Implement concurrent query serving: ingestion changes and the query router #865
• Serve the API from the v2 stores (support both v1 and v2) — 🟡
  • Wire the v2 stores into the API handlers and support both v1 and v2 #772
  • shared JSON-RPC handlers over store interfaces; one repo, two binaries; v1 stays through a deprecation window
  • ends with feature/full-history merged to main and retired
• Events endpoint v2 — 🟡
  • ✅ Revisit getEvents API #426
  • Implement the v2 events endpoint and deprecate the existing getEvents #774

Validation (trust-min) — ⬜ · P2

When complete: ingested data is verified correct without trusting the data-lake source.

• Validate ingested data — trust-minimize the backfill source #773

Benchmarks — 🟡

When complete: perf is measured consistently and regressions are caught.

• Run the Phase 1–3 ingestion benchmarks for the hot DB — ✅ · P0 — Phase 1 passes all profiles; the Phase 2/3 misses drive Investigate and fix the issues surfaced by the O3 ingestion load test #800
  • ✅ Synthetic ledger datasets for Phases 1–3 ("golden packs") #867
  • ✅ Run the Phase 1 benchmark campaign — ingest and reads at 2 s blocks #872
  • ✅ Run the Phase 2 benchmark campaign — ingest and reads at 1 s blocks #873
  • ✅ Run the Phase 3 benchmark campaign — ingest and reads at 600 ms blocks #874
• Experiments and writeup to achieve the Phase 1–3 ingestion targets — 🟡 · P0
  • Investigate and fix the issues surfaced by the O3 ingestion load test #800
• Benchmark commands — 🟡 · P0
  • ✅ Full-history benchmark commands — ingest across all data types #771
  • Full-history query benchmark commands — reads across all data types  #856
• Benchmark results explorer: browsable reports for campaign runs — ✅ · P1
  • Benchmark results explorer — browsable reports for campaign runs #871
• Benchmark transaction-submission latency — ✅ · P1
  • Benchmark transaction-submission latency #869
• Full-history backfill benchmark: time + disk footprint on minimum hardware — ✅ · P1
  • Benchmark full-history backfill — time and disk footprint on minimum hardware #895
• Obtaining results: benchmark RPC v2 DB layer performance against real pubnet activity and O3 ledger profiles — ✅ · P1
  • bench(fullhistory): 2026-05-21 cross-machine results report #750
  • bench(fullhistory): synthetic-ledger datasets via stellar-core apply-load as an ingest/query bench source #762
• Automate benchmark runs in CI — 🟡 · P2
  • e2e benchmark campaigns workflow — dispatch, run on EC2, publish results #868
  • Run full-history benchmark commands in CI to catch perf regressions #775
• Benchmarking infra: S3 bucket + benchmarking devbox — ✅ · P2
  • Benchmarking infra: stellar-rpc-bench S3 bucket + benchmarking devbox #866
• Read SLAs under ingestion load — ⬜ · P2
  • Read-endpoint SLAs under ingestion load #870

EBS / historical tier — ⬜ · P3

When complete: retention can extend beyond local NVMe by spilling older immutable files to EBS, served at higher latency.

• (no issue yet)

📅 MVP target

Best ~Sep 15 2026 (end of sprint 75) · worst ~Oct 13 2026 (end of sprint 76) for the MVP: query serving complete — 4-week sprints. Estimates, not commitments.

Where we are. The query router is implemented (#865). What remains for MVP is the back half of serving the API from the v2 stores (#772: the store adapters, handler wiring, and the parity suite; its foundations are landed) and the v2 events API (#774: cursor + codec landed).

Remaining critical path (from sprint 75 — the two remaining pieces run in parallel under different owners; with the router landed, #772 is no longer gated on anything):

Step	Best	Worst	When
Serve the API from the v2 stores (#772)	1	2	adapters → wiring → parity; merge to main lands last
v2 events API + v1 support (#774)	1	2	sprint 75 on, in parallel

• Best (→ ~Sep 15): both land in sprint 75.
• Worst (→ ~Oct 13): a piece slips into sprint 76.

⚠️ Risks

What could push the date out — largest first:

1. Live ingestion design — the most complex piece to design, and the sprint-73 focus (design + implementation in the same sprint). If the design runs long, the whole unified ingestion workflow — and the timeline — slips with it.
2. Automated, repeatable benchmarking — running benchmarks in an automated, repeatable way to detect regressions is something we've never built. The approach is unproven, so its effort is the hardest to estimate.
3. Query serving design — Design: concurrent query serving across all data types #770 is also a sprint-73 item and isn't settled yet; query-serving implementation is gated on it.
4. Team capacity — sprint 75 again runs several big pieces in parallel (Wire the v2 stores into the API handlers and support both v1 and v2 #772's remaining half, Implement the v2 events endpoint and deprecate the existing getEvents #774, Investigate and fix the issues surfaced by the O3 ingestion load test #800) across a small team. If we can't sustain that parallelism, work serializes and the date slips.
5. Emergent work — unplanned interrupts (e.g. HackerOne reports against RPC or Horizon) preempt sprint work on short notice. They are unpredictable in timing and size, and they land on the same small team that holds the critical path, so a bad one stalls the sprint.

[tamirms]

Sprint 72 (ended Jun 23)

• DB layer largely landed — events core (events,eventstore: concurrent index; migrate off membitmaps #756), events query engine ([events] Single-chunk query engine #663), txhash cold store (TxHash cold store — streamhash index (build + read) #728), XDR view extractors (XDR view extractors: events, tx-hashes, tx-details, tx-pages from LedgerCloseMetaView #764), ingestion (Ingestion: all data types for both cold and hot stores #765).
• Benchmark results for the DB layer vs. real pubnet + O3 profiles (bench(fullhistory): 2026-05-21 cross-machine results report #750, bench(fullhistory): synthetic-ledger datasets via stellar-core apply-load as an ingest/query bench source #762).