Skip to content

feat(indexer): vendor squid-subsquid-network into the monorepo - #79

Open
belopash wants to merge 146 commits into
developfrom
cursor/real-indexer-local-setup-1795
Open

feat(indexer): vendor squid-subsquid-network into the monorepo#79
belopash wants to merge 146 commits into
developfrom
cursor/real-indexer-local-setup-1795

Conversation

@belopash

Copy link
Copy Markdown
Contributor

Summary

Merges the subsquid/squid-subsquid-network indexer into this monorepo via git subtree. After this PR the four production squids (workers, gateways, token, vestings) plus their shared package live as first-class workspace members under packages/, and all workspace packages — app and indexer — share the @sqd/* npm scope.

Why

  • Single source of truth — address-book / ABI / schema changes happen in one PR, not two.
  • Drops the build-time tools/devnet/patches/ step (follow-up PR after the companion devnet PR lands).
  • Makes type-level coupling between client/server and the indexer's GraphQL schema tractable in the future.

Layout after this PR

packages/
├── client/             # @sqd/client          (renamed from @subsquid/client)
├── common/             # @sqd/common          (renamed from @subsquid/common)
├── server/             # @sqd/server          (renamed from @subsquid/server)
├── indexer-common/     # @sqd/indexer-common  (renamed from @sqd/shared, vendored)
├── indexer-workers/    # @sqd/workers         (vendored)
├── indexer-gateways/   # @sqd/gateways        (vendored)
├── indexer-token/      # @sqd/token           (vendored)
└── indexer-vestings/   # @sqd/vestings        (vendored)
indexer/                # vendored squid tooling: ABIs, manifests, scripts, plans, biome.json, tsconfig.base.json

Key decisions

  • git subtree add, no --squash. Preserves upstream history. Future bumps: git subtree pull --prefix=indexer https://github.com/subsquid/squid-subsquid-network.git master (then re-flatten + re-rename).
  • Flat layout under packages/. Indexer packages live alongside app packages so a single workspace glob (packages/*) covers everything.
  • indexer-common rename. The vendored @sqd/shared is renamed to @sqd/indexer-common to make the role obvious next to the existing app-side @sqd/common. There IS conceptual overlap between the two (both ship contract ABIs + address books for the same contracts), but the shape differs — wagmi-tuple vs. @subsquid/evm-codec event/fun factories — so deduplication needs a real API unification. Tracked as a follow-up; deliberately out of scope of this PR.
  • Scope unification. @subsquid/{client,common,server}@sqd/{client,common,server}. @subsquid/* references in source/imports now refer exclusively to upstream npm packages (@subsquid/batch-processor, @subsquid/typeorm-store, etc.).

Toolchain coexistence

App Indexer
TypeScript 5.9 5.9 (verified — indexer builds clean against 5.9; decorator metadata works)
Vitest 4.x 2.x (per-package, no conflict)
Biome 2.3 1.9 (root biome ignores indexer/ and packages/indexer-*; indexer keeps its own indexer/biome.json)
Module type ESM CJS (per-package; rolldown bundles to CJS)
Output dir build/, dist/ lib/ (added to root .gitignore)

Verification

  • pnpm install succeeds; lockfile committed.
  • pnpm build — full turbo graph (8 packages) green: client / common / server / 5× indexer-*.
  • pnpm build:indexer — runs rolldown -c && tsc -p tsconfig.model.json for each indexer-* package.
  • pnpm lint — green (3 pre-existing client warnings, unrelated).
  • pnpm tsc — green for @sqd/client and @sqd/server.
  • pnpm --filter @sqd/workers test — 6 vitest tests pass under indexer-bundled vitest 2.

Out of scope (explicit non-goals)

  • ABI deduplication between @sqd/common and @sqd/indexer-common — needs API unification first.
  • ESM migration of indexer packages.
  • Unifying biome / vitest / typescript major versions across both stacks.
  • CI to build/deploy the indexer from this repo. The indexer is deployed via SQD's own pipeline; this repo only vendors source.
  • The companion devnet-stack PR (cursor/mock-wallet-d5dd) — once both land, a small follow-up drops external/squid-subsquid-network submodule + tools/devnet/patches/ + the Dockerfile copy step.

Follow-ups

  1. Drop the indexer submodule + devnet build-time patch (after both PRs land).
  2. ABI / address-book deduplication between @sqd/common and @sqd/indexer-common.
  3. (Optional) CI for indexer builds in this repo.
Open in Web Open in Cursor 

belopash and others added 29 commits April 15, 2026 17:32
Use ownerId (plain string) instead of FK owner relation in workers and
gateways schemas. Reorganize gateways handlers into gateway/ and pool/
subdirectories and merge portal-pools logic. Add MV refresh loop to
workers and token main.ts. Batch stats log changed to debug level.

Made-with: Cursor
Add per-indexer server-extension resolvers with timeseries queries for
workers, token, and gateways. Add materialized view refresh modules
(hourly via setTimeout) and standalone refresh scripts for workers and
token. Generate database migrations for all indexers.

Made-with: Cursor
Move deployment manifests from packages/*/manifests/ to manifests/
at the repo root for monorepo-wide sqd deploy. Update manifest commands
to use pnpm --dir for correct working directory. Remove old monolithic
refresh script and grafana dashboard.

Made-with: Cursor
…tch-up

- Add PoolProvider entity to gateways indexer to track per-provider
  deposits on portal pools
- Add PoolCreated handler to token indexer so portal pool template
  is registered and deposit/withdraw transfers are classified
- Fix workers MV migration ordering (must run after Data migration)
- Make epoch tracking catch up missed epochs in a loop instead of
  requiring includeAllBlocks

Made-with: Cursor
- Batch WorkerReward find() in recalculateWorkerAprs (cap.ts) to single query
- Replace per-worker WorkerMetrics N+1 with SQL-side GROUP BY aggregation
- Batch Delegation find() in Distributed.handler.ts, make distributeReward sync
- Fix blocksPassed=Infinity bug causing pruning on first batch
- Use keyset pagination for block pruning instead of OFFSET
- Start processor from network.range.from to capture EpochLengthUpdated event

Made-with: Cursor
…ndlers

Remove Staking.Rewarded and RewardTreasury.Claimed event subscriptions
that had no corresponding handlers. Replace hard assert with warning log
in DistributedRewardsDistribution Claimed handler to prevent indexer
crash on unexpected claimer addresses. Fix missing worker relation in
Distributed handler delegation query that caused "Cannot read properties
of undefined (reading 'id')" errors. Document lock period assumptions
and reward math divergence from on-chain contracts.

Made-with: Cursor
- Add shared async-task helper, group size tweaks, and rpc-client update\n- Refactor cap handlers; add softcap handler and StatsChunkCursor migration\n- Extend workers metrics pipeline and align timeseries resolvers (gateways, token, workers)\n- Add vitest setup and cap handler tests; update manifests and toolchain (pnpm, turbo)

Made-with: Cursor
split indexers into multiple instances
…regates

Previously, uptime was averaged only over hours that had a recorded row,
so fully-offline hours were silently excluded and inflated the percentage.
Now the denominator is the total elapsed hours in the window since the
later of the window start or the worker's registration date, so missing
hours count as zero uptime.

Also initialise aggregatesPending=true so aggregates are always
recomputed on the first caught_up after startup, ensuring stale values
are refreshed immediately after a code deployment even when no new stats
chunks are pending.

Made-with: Cursor
Change the daily uptime materialized view from AVG(uptime) — which only
averages hours that have a recorded row and ignores offline gaps — to
SUM(uptime) / 24.0, so that hours with no data are treated as 0%
uptime. This makes the per-day chart consistent with the corrected 90d
aggregate calculation.

Made-with: Cursor
Previously lastMetricsFetchAt was set when a new burst started. If the
burst completed (caught_up) quickly — e.g. only a few new chunks to
process — the 30-minute cooldown had almost entirely elapsed before
caught_up fired, so the next poll was delayed by nearly a full interval
from that point. With new hourly chunks on the server the squid appeared
to stop checking after printing "applied all available chunks up to…".

Now lastMetricsFetchAt is set at caught_up (and on a null/failed fetch),
so the 30-minute window always starts from when a fetch cycle actually
completes. The burst itself remains unthrottled (metricsBurstActive
bypasses the check), and the startup burst fires immediately as before
because lastMetricsFetchAt starts at -1.

Made-with: Cursor
…ution

Distributed events cover a past block range and can legitimately credit a
worker that withdrew during or after that range. The contract still allows
the former operator to claim those rewards, so filter only on nullness, not
on status.

Made-with: Cursor
git-subtree-dir: indexer
git-subtree-mainline: 250e3bc
git-subtree-split: 9b1f5c3
…ockfile/workspace/turbo/npmrc/docker-compose)

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…ignore from root biome

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
… (shared\u2192indexer-common)

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…rack indexer lib/ output

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…/* (npm names + imports); biome ignore packages/indexer-*

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…nc workflow

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
…exer-common imports from it

- Convert @sqd/common to CJS: drop type:module, tsc emits lib/ as CommonJS,
  package.json exports point at lib/ for runtime + types.
- Add 4 indexer-only addresses to @sqd/common (NETWORK_CONTROLLER,
  REWARD_CALCULATION, VESTING_FACTORY, TEMPORARY_HOLDING_FACTORY) and
  switch MULTICALL to canonical Multicall3 (0xca11bde0\u2026 — was unused
  by app code).
- Add @sqd/common as workspace dep of @sqd/indexer-common.
- Rewrite indexer-common's network.ts to derive contracts +
  defaultRouterContracts from CONTRACT_ADDRESSES; keep indexer-specific
  block-number metadata (range.from, epochsStart, softCap.from) inline.
- Set turbo dev.dependsOn=^build so tsc lib/ output exists before
  consumers run.

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
- Move indexer/{abi,assets,manifests,plans,scripts,tsconfig.base.json,.env.indexer.example}
  to repo root. plans/ and scripts/ get prefixed (indexer-plans, indexer-scripts) to
  avoid colliding with future app-side equivalents.
- Delete duplicates: indexer/biome.json (root biome covers everything; vendored
  packages/indexer-* still excluded), indexer/.gitignore (root covers lib/),
  indexer/.gitmodules + empty subsquid-network-contracts/ (sqd-portal-contracts
  is the only submodule we keep), indexer/docker (create-databases.sh superseded
  by tools/devnet/), indexer/LICENSE + README.md + .squidignore (deploy-pipeline
  artifacts unused in this repo), indexer/.agents (turborepo+typescript skills
  duplicated by root .agents/).
- Repoint indexer-*/tsconfig{,.model}.json extends path:
    ../../indexer/tsconfig.base.json \u2192 ../../tsconfig.indexer.base.json
- Fix squid manifest paths: pnpm --dir packages/<pkg> \u2192 packages/indexer-<pkg>
- Drop redundant 'indexer/packages/*' glob from pnpm-workspace.yaml.
- Update biome.json ignore globs and AGENTS.md/README.md repository-tour blocks.

Co-authored-by: Alexander Belopashentsev <belopash@users.noreply.github.com>
@belopash
belopash marked this pull request as ready for review April 27, 2026 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant