feat(indexer): vendor squid-subsquid-network into the monorepo - #79
Open
belopash wants to merge 146 commits into
Open
feat(indexer): vendor squid-subsquid-network into the monorepo#79belopash wants to merge 146 commits into
belopash wants to merge 146 commits into
Conversation
Use ownerId (plain string) instead of FK owner relation in workers and gateways schemas. Reorganize gateways handlers into gateway/ and pool/ subdirectories and merge portal-pools logic. Add MV refresh loop to workers and token main.ts. Batch stats log changed to debug level. Made-with: Cursor
Add per-indexer server-extension resolvers with timeseries queries for workers, token, and gateways. Add materialized view refresh modules (hourly via setTimeout) and standalone refresh scripts for workers and token. Generate database migrations for all indexers. Made-with: Cursor
Move deployment manifests from packages/*/manifests/ to manifests/ at the repo root for monorepo-wide sqd deploy. Update manifest commands to use pnpm --dir for correct working directory. Remove old monolithic refresh script and grafana dashboard. Made-with: Cursor
…tch-up - Add PoolProvider entity to gateways indexer to track per-provider deposits on portal pools - Add PoolCreated handler to token indexer so portal pool template is registered and deposit/withdraw transfers are classified - Fix workers MV migration ordering (must run after Data migration) - Make epoch tracking catch up missed epochs in a loop instead of requiring includeAllBlocks Made-with: Cursor
Made-with: Cursor
- Batch WorkerReward find() in recalculateWorkerAprs (cap.ts) to single query - Replace per-worker WorkerMetrics N+1 with SQL-side GROUP BY aggregation - Batch Delegation find() in Distributed.handler.ts, make distributeReward sync - Fix blocksPassed=Infinity bug causing pruning on first batch - Use keyset pagination for block pruning instead of OFFSET - Start processor from network.range.from to capture EpochLengthUpdated event Made-with: Cursor
…ndlers Remove Staking.Rewarded and RewardTreasury.Claimed event subscriptions that had no corresponding handlers. Replace hard assert with warning log in DistributedRewardsDistribution Claimed handler to prevent indexer crash on unexpected claimer addresses. Fix missing worker relation in Distributed handler delegation query that caused "Cannot read properties of undefined (reading 'id')" errors. Document lock period assumptions and reward math divergence from on-chain contracts. Made-with: Cursor
- Add shared async-task helper, group size tweaks, and rpc-client update\n- Refactor cap handlers; add softcap handler and StatsChunkCursor migration\n- Extend workers metrics pipeline and align timeseries resolvers (gateways, token, workers)\n- Add vitest setup and cap handler tests; update manifests and toolchain (pnpm, turbo) Made-with: Cursor
split indexers into multiple instances
…regates Previously, uptime was averaged only over hours that had a recorded row, so fully-offline hours were silently excluded and inflated the percentage. Now the denominator is the total elapsed hours in the window since the later of the window start or the worker's registration date, so missing hours count as zero uptime. Also initialise aggregatesPending=true so aggregates are always recomputed on the first caught_up after startup, ensuring stale values are refreshed immediately after a code deployment even when no new stats chunks are pending. Made-with: Cursor
Change the daily uptime materialized view from AVG(uptime) — which only averages hours that have a recorded row and ignores offline gaps — to SUM(uptime) / 24.0, so that hours with no data are treated as 0% uptime. This makes the per-day chart consistent with the corrected 90d aggregate calculation. Made-with: Cursor
Previously lastMetricsFetchAt was set when a new burst started. If the burst completed (caught_up) quickly — e.g. only a few new chunks to process — the 30-minute cooldown had almost entirely elapsed before caught_up fired, so the next poll was delayed by nearly a full interval from that point. With new hourly chunks on the server the squid appeared to stop checking after printing "applied all available chunks up to…". Now lastMetricsFetchAt is set at caught_up (and on a null/failed fetch), so the 30-minute window always starts from when a fetch cycle actually completes. The burst itself remains unthrottled (metricsBurstActive bypasses the check), and the startup burst fires immediately as before because lastMetricsFetchAt starts at -1. Made-with: Cursor
Made-with: Cursor
Made-with: Cursor
…ution Distributed events cover a past block range and can legitimately credit a worker that withdrew during or after that range. The contract still allows the former operator to claim those rewards, so filter only on nullness, not on status. Made-with: Cursor
Made-with: Cursor
…ockfile/workspace/turbo/npmrc/docker-compose) Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…ignore from root biome Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
… (shared\u2192indexer-common) Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…rack indexer lib/ output Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…/* (npm names + imports); biome ignore packages/indexer-* Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…nc workflow Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…exer-common imports from it - Convert @sqd/common to CJS: drop type:module, tsc emits lib/ as CommonJS, package.json exports point at lib/ for runtime + types. - Add 4 indexer-only addresses to @sqd/common (NETWORK_CONTROLLER, REWARD_CALCULATION, VESTING_FACTORY, TEMPORARY_HOLDING_FACTORY) and switch MULTICALL to canonical Multicall3 (0xca11bde0\u2026 — was unused by app code). - Add @sqd/common as workspace dep of @sqd/indexer-common. - Rewrite indexer-common's network.ts to derive contracts + defaultRouterContracts from CONTRACT_ADDRESSES; keep indexer-specific block-number metadata (range.from, epochsStart, softCap.from) inline. - Set turbo dev.dependsOn=^build so tsc lib/ output exists before consumers run. Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
- Move indexer/{abi,assets,manifests,plans,scripts,tsconfig.base.json,.env.indexer.example}
to repo root. plans/ and scripts/ get prefixed (indexer-plans, indexer-scripts) to
avoid colliding with future app-side equivalents.
- Delete duplicates: indexer/biome.json (root biome covers everything; vendored
packages/indexer-* still excluded), indexer/.gitignore (root covers lib/),
indexer/.gitmodules + empty subsquid-network-contracts/ (sqd-portal-contracts
is the only submodule we keep), indexer/docker (create-databases.sh superseded
by tools/devnet/), indexer/LICENSE + README.md + .squidignore (deploy-pipeline
artifacts unused in this repo), indexer/.agents (turborepo+typescript skills
duplicated by root .agents/).
- Repoint indexer-*/tsconfig{,.model}.json extends path:
../../indexer/tsconfig.base.json \u2192 ../../tsconfig.indexer.base.json
- Fix squid manifest paths: pnpm --dir packages/<pkg> \u2192 packages/indexer-<pkg>
- Drop redundant 'indexer/packages/*' glob from pnpm-workspace.yaml.
- Update biome.json ignore globs and AGENTS.md/README.md repository-tour blocks.
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
belopash
marked this pull request as ready for review
April 27, 2026 20:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Merges the
subsquid/squid-subsquid-networkindexer into this monorepo viagit subtree. After this PR the four production squids (workers,gateways,token,vestings) plus their shared package live as first-class workspace members underpackages/, and all workspace packages — app and indexer — share the@sqd/*npm scope.Why
tools/devnet/patches/step (follow-up PR after the companion devnet PR lands).Layout after this PR
Key decisions
git subtree add, no--squash. Preserves upstream history. Future bumps:git subtree pull --prefix=indexer https://github.com/subsquid/squid-subsquid-network.git master(then re-flatten + re-rename).packages/. Indexer packages live alongside app packages so a single workspace glob (packages/*) covers everything.indexer-commonrename. The vendored@sqd/sharedis renamed to@sqd/indexer-commonto make the role obvious next to the existing app-side@sqd/common. There IS conceptual overlap between the two (both ship contract ABIs + address books for the same contracts), but the shape differs — wagmi-tuple vs.@subsquid/evm-codecevent/fun factories — so deduplication needs a real API unification. Tracked as a follow-up; deliberately out of scope of this PR.@subsquid/{client,common,server}→@sqd/{client,common,server}.@subsquid/*references in source/imports now refer exclusively to upstream npm packages (@subsquid/batch-processor,@subsquid/typeorm-store, etc.).Toolchain coexistence
indexer/andpackages/indexer-*; indexer keeps its ownindexer/biome.json)build/,dist/lib/(added to root.gitignore)Verification
pnpm installsucceeds; lockfile committed.pnpm build— full turbo graph (8 packages) green: client / common / server / 5× indexer-*.pnpm build:indexer— runsrolldown -c && tsc -p tsconfig.model.jsonfor each indexer-* package.pnpm lint— green (3 pre-existing client warnings, unrelated).pnpm tsc— green for@sqd/clientand@sqd/server.pnpm --filter @sqd/workers test— 6 vitest tests pass under indexer-bundled vitest 2.Out of scope (explicit non-goals)
@sqd/commonand@sqd/indexer-common— needs API unification first.cursor/mock-wallet-d5dd) — once both land, a small follow-up dropsexternal/squid-subsquid-networksubmodule +tools/devnet/patches/+ the Dockerfile copy step.Follow-ups
@sqd/commonand@sqd/indexer-common.