New check: retrieval++

[BigLep]

Done Criteria

We have a retrieval check that does not easily reveal the checker as dealbot by:

1. Running on generic/dynamic infrastructure (rather than just coming from static dealbot k8s clusters).
2. Checking for retrieval of pieces beyond what dealbot created. Anything on chain with an ipfsRootCid is fair game.

Once a piece for checking has been determined, it should follow similar behavior to the basic retrieval check of doing CID looking in IPNI and doing /ipfs retrieval.

All metrics should have similar dimensionality to basic retrieval check (e.g., provider info, checkType). A new check type should be used to disambiguate this from the basic retrieval check.

Open questions

1. What to call this new checkType (e.g., retrievalAnon)?
2. Piece selection logic/method (e.g., query subgraph X for all pieces with ipfsRootCid on provider X)
3. How much of the piece is downloaded (e.g., download the whole piece, jus the first block, just the first X Mb)?
4. Do we make these "anonymous retrieval checks" form all locations to all SPs or be more geographically aware (e.g., check on EU SPs from the EU). We should follow suit or learn from Support checks from multiple regions #246

Why Important

This gives a more honest analysis of how an SP is doing. This methodology makes it harder for the SP to be on their best behavior because "the teacher is watching".

User/Customer

Propsective/current FOC users who want to have confidence that data stored with FOC will be truly retrievable later by anyone.

Notes

[rvagg]

Things we care about:

• Piece retrieval works for whole piece and the bytes are correct
• IPFS retrieval works for all of the blocks in a CAR that we upload in a deal with withIPFSIndexing on
• IPNI has all of the CIDs in a CAR in a piece stored with withIPFSIndexing on

We also need to be careful not to blame the SP for data that isn't actually IPFS data but the client claims it is.

Here's how I propose to handle this:

1. Find pieces to fetch. They're going to want to be large enough that you get decent bandwidth metrics from them, but not so large as to exclude providers who don't happen to have big deals or you're testing the same pieces regularly. Maybe we have to be fully open about size criteria but account for it (somehow) in any speed testing.

Here's the distribution of piece size on both networks currently

But we'd also want to focus on pieces in data sets that have the withIPFSIndexing metadata set. This is what tells the SP to index the pieces if they are CARs and serve those IPLD blocks through the ipfs retrieval endpoint. Without that metadata, we can't expect a piece that's a CAR to be retrievable, we've made it opt-in (unlike classic PoRep).

Here's the piece / size distribution accounting for withIPFSIndexing

We could randomly sample from both withIPFSIndexing and non-withIPFSIndexing pools just to get full coverage (to ensure an SP isn't prioritising the CAR pieces), but we do want to make sure we have a good representation of withIPFSIndexing pieces in our pool.

So, this is probably going to need a subgraph or other indexer of FOC (I'm running https://github.com/filozone/foc-observer, but a subgraph specific to this task might be better). It would need to collect data sets, data set metadata, piece CID, piece size (contained in the Piece CID so you can decode it in the subgraph code) and provider details (you can map data set owner to provider wallet to get provider ID in the ServiceProviderRegistry).

2. Given a piece, perform a full retrieval via the /piece/{pieceCID} endpoint for the SP, measure speed, latency etc. Then hash the piece with CommP and make sure it matches the PieceCID you have. If it fails, this is a heavy black mark against the SP.

3. If the piece is in a data set with withIPFSIndexing check whether the piece is a CAR, if it's a CAR, inspect it, make sure it's valid and the block data match the CIDs and get the list of CIDs in it (go-car has both car ls and car inspect --full that can show how to do this, also possible in JS). If the piece isn't a CAR or the CAR isn't valid in any way, don't proceed, and it's not the SP's fault, this is on the client.

4. Check IPNI for all of those CIDs in the CAR, does IPNI have records that link the individual CIDs back to the provider. If not, that's a mark against the SP.

5. Perform an /ipfs/{ipfsCID} retrieval for some or all of the CIDs in the CAR against the SP and then hash the returned bytes against the CID to make sure it's correct. Any failure to serve or bytes that don't hash properly are a mark against the SP.

Things we can't assume:

• Client uploaded a CAR in a withIPFSIndexing data set
• Client uploaded a valid CAR (SP will try to index but bail if it's not valid)
• CAR is a coherent DAG of blocks with a valid root; there may be no root in the CAR header and the blocks may be a random assortment with no links between them (filecoin-pin add <path> will make a complete DAG, put a root in the CAR and put ipfsRootCID on chain metadata for the piece, but this isn't strictly required or expected)
• CAR contains blocks of certain sizes--technically there are only some basic restrictions in go-car, but even the spec itself doesn't place requirements, so you may even find a block that's a CAR with a single 1GiB IPLD block in it that's still valid.
• Piece size will be nicely bounded. There's a technical limitation on CommP at the lower end, so don't expect smaller than 127 bytes, but upper bound is not strict. In Curio we cap it to 1016MiB but we may uncap that and we also aren't necessarily just seeing SPs using Curio and our restrictions, Storacha is involved and we may see custom forks or implementations. PDP & FWSS don't strictly limit size.

[BigLep]

Fantastic write up @rvagg! I'm fully on board with this proposal. I'll get the done criteria updated 2026-04-02 to match this.

This seems like a great new check to add that give higher assurances that SPs are serving retrievals of all stored data, not just dealbot data.