937 follow-ups: getTransactions cursor guards, getHealth first-commit gate, snapshot gauge, method-table completeness test - #953
Conversation
snapRefs only spoke at teardown; a leaked read view was invisible while the process ran. A package-level tally mirrors deferredCloseOps and observability exports it as open_snapshots.
MethodsConfig fields drive the check: limitsByMethod must key every method, and validateService must reject a zero queue limit on each. Apply already ties limits to the served specs, so the tables cannot drift apart silently.
A caught-up poller used to get the zero cursor "0" (then -32602 on every retry, #745) or a regressed cursor that re-delivered a consumed ledger's transactions. The walk's result is now floored at the request's cursor, so the returned token always fetches what comes next. Shared handler: changes v1 wire behavior too; CHANGELOG says so.
Close times survive restarts, so a restarted node reported healthy off the previous run's commit for up to max_healthy_ledger_latency. The registry records the boot-seeded latest ledger; getHealth fails until ingestion advances past it. v1 keeps the shared behavior.
There was a problem hiding this comment.
Pull request overview
Adds follow-ups for pagination correctness, startup health gating, snapshot observability, and method-table completeness.
Changes:
- Prevents
getTransactionscursor regression. - Gates v2 health until the first post-boot ingestion commit.
- Adds snapshot metrics and method-configuration completeness tests.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
CHANGELOG.md |
Documents cursor behavior changes. |
methods/get_transactions.go |
Floors responses at the requested cursor. |
methods/get_transactions_test.go |
Tests caught-up cursor handling. |
rpcv2/jsonrpc.go |
Adds the health startup gate. |
rpcv2/jsonrpc_test.go |
Tests pre/post-commit health responses. |
rpcv2/methods_completeness_test.go |
Verifies method-table coverage. |
rpcv2/observability/observability.go |
Exports the snapshot gauge. |
rpcv2/observability/observability_test.go |
Verifies gauge registration. |
rpcv2/query/registry.go |
Tracks the boot ledger sequence. |
rpcv2/query/registry_test.go |
Tests commit-since-boot detection. |
rpcv2/rocksdb/rocksdb.go |
Tracks unreleased snapshots globally. |
rpcv2/rocksdb/snapshot_test.go |
Tests snapshot gauge accounting. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // produces nothing and leaves the cursor below the request's own — at the | ||
| // zero value, or at the consumed ledger's start. Echo the request's cursor | ||
| // instead: the returned token must always fetch what comes next (#745). | ||
| if requestCursor != nil && cursor.ToInt64() < requestCursor.ToInt64() { |
There was a problem hiding this comment.
While verifying the cursor guard here I checked how the other two paginated methods behave for the same caught-up poll. getEvents is fine: an empty page returns a max cursor at the last scanned ledger, which stays valid when resent at the tip. getLedgers still breaks the polling loop, and it fails with an error instead of an empty page:
- A poller reaches the tip ledger T. getLedgers returns its last page with
cursor: "T". - The poller resends
cursor: "T".parseCursorcomputesstart = T + 1and rejects it, becauseIsLedgerWithinRangerequiresstart <= T(get_ledgers.go, line 154). - The poller gets
-32602: "cursor ('T') must be between the oldest ledger: X and the latest ledger: T". Every poll fails until the next ledger closes.
This self-heals within one ledger interval, so nobody gets permanently stuck the way #745 clients did. But it is the steady state for anyone polling faster than the close interval, and the code is the same -32602 a malformed cursor gets, so a client cannot tell "wait and retry" apart from "my cursor is garbage" without parsing the message text. The handler already contains the intended behavior: the empty-result branch echoes request.Pagination.Cursor (line 106), but validation rejects the caught-up request before that branch can run, so it is unreachable for exactly the case it was written for.
Since the changelog entry in this PR states the contract generally ("the returned cursor now always does what the docs promise"), I think we should bring getLedgers in line, either as one more commit here or as an immediate follow-up, ideally landing in the same release so the caught-up contract changes once for both methods. The shape I have in mind mirrors the getTransactions fix: parseCursor keeps only the lower-bound rejection (a cursor below the oldest ledger is real data loss and should stay an error), and the handler returns early for a cursor at or past the tip, before end is computed or fetchLedgers runs:
// A caught-up poller's cursor points at or past the tip. Echo it back on an
// empty page instead of rejecting the server's own token.
if request.Pagination != nil && request.Pagination.Cursor != "" &&
start > availableLedgerRange.LastLedger {
return protocol.GetLedgersResponse{
Ledgers: []protocol.LedgerInfo{},
LatestLedger: ledgerRange.LastLedger.Sequence,
LatestLedgerCloseTime: ledgerRange.LastLedger.CloseTime,
OldestLedger: ledgerRange.FirstLedger.Sequence,
OldestLedgerCloseTime: ledgerRange.FirstLedger.CloseTime,
Cursor: request.Pagination.Cursor,
}, nil
}Two details worth pinning in tests: an explicit startLedger above the tip should stay an error, the same asymmetry getTransactions has (only the server-issued token gets the echo, an explicit out-of-range start is a client mistake), and the cursor "4294967295" makes start wrap to 0 in parseCursor, which the retained lower-bound check needs to catch. Placing the early return before end := start + uint32(limit) - 1 also keeps that addition from ever running with an above-tip start.
If you'd rather defer this, a row on #937 plus scoping the changelog sentence to getTransactions would keep the docs honest in the meantime.
There was a problem hiding this comment.
Fair callout — done in 37d8424, making getLedgers toe the same cursor-semantics line as getTransactions.
Resending the last page's cursor at the tip failed with the same -32602 a malformed cursor gets, until the next ledger closed. The cursor parse now rejects only the lower bound (below the oldest ledger is data loss); at or past the tip returns an empty page with the cursor echoed, matching getTransactions. A max-uint32 cursor wraps to start 0 and stays rejected.
Executes the remaining #937 items against the current branch tip. Four items land here; H2 stays deferred in #937.
What changed
open_snapshotsgauge: a process-wide tally of unreleased RocksDB snapshots, exported beside the serving-invariant counters. A leaked read view is visible on a dashboard, not only in a teardown logrpcv2/rocksdb/rocksdb.go,rpcv2/observability/observability.goMethodsConfigmethod must appear inlimitsByMethodand be checked byvalidateService.Applyalready ties limits to the served specsrpcv2/methods_completeness_test.go(test-only)getTransactionsnever returns a cursor below the request's cursor. A caught-up poller gets its cursor echoed back instead of"0"(#745) or a regressed cursor that re-delivers a consumed ledgermethods/get_transactions.go,CHANGELOG.mdgetLedgersgets the same caught-up contract: a cursor at or past the tip returns an empty page with the cursor echoed, instead of-32602. The cursor parse keeps only the lower-bound rejection; a max-uint32 cursor wraps to start 0 and stays rejectedmethods/get_ledgers.go,CHANGELOG.mdgetHealthfails until this run's ingestion commits its first ledger. The registry records the boot-seeded latest ledger; the gate compares against itrpcv2/query/registry.go,rpcv2/jsonrpc.goWire-visible changes
getTransactions(v1 AND v2 — shared handler)"0"; resending it failed every poll with-32602. A consumed ledger's cursor regressed and the next poll re-delivered its transactionsgetLedgers(v1 AND v2 — shared handler)-32602— the same code a malformed cursor gets — until the next ledger closedstartLedgerstill error. CHANGELOG carries a loud entrygetHealth(v2 only)max_healthy_ledger_latencyafter a restart, off the previous run's close time — even in a crash loop-32603until the first commit of this run lands (about one ledger interval after boot). v1 unchangedNot in this PR
.idxreader cache)Notes
open_snapshotsmirrors thedeferredCloseOpspattern: a package atomic read by aGaugeFunc.MethodsConfig: adding a method field extends the checks with no test edits.