Skip to content

Proposal: gRPC based indexer #83

@linuskendall

Description

@linuskendall

Currently the DAS API stack relies on having a geyser plugin indexer. One proposal would be to add/transition to a geyser grpc based indexer (https://github.com/rpcpool/yellowstone-grpc). This would allow:

  1. Multiple parallel indexers connected to different grpc endpoints (even different providers)
  2. Better failover
  3. Lower cost since Yellowstone gRPC can be delivered as a shared service.

For account updates and transaction updates this should be pretty straight forward, you can subscribe to confirmed or finalized commitment level via gRPC, convert to Plerkle format and push to redis.

You could also implement a block check mode which checks that you have indexed all transactions from a given block and if NOT, fetch that block via gRPC to fill in missing TXs.

There is experimental support in yellowstone-grpc for fetching program account state which would allow snapshots of the Tokenkeg account to be taken for indexing of traditional NFTs. Snapshots could also be handled through one time ingest through a tool like https://github.com/terorie/solana-snapshot-etl which could even run the plerkle plugin to update all accounts at once. With snapshot-etl the order would be:

  1. Start up the geyser grpc ingestion
  2. Run snapshot-etl to fill in all the missing assets while grpc subscriptions are running.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions