feat(dag): add --local-only to dag export and import by ChayanDass · Pull Request #11229 · ipfs/kubo

ChayanDass · 2026-03-07T08:01:17Z

Add `--local-only` to `dag export` and `dag import`

Summary

Adds support for partial CAR export and import: export only blocks that exist locally (skipping missing blocks), and import such partial CARs without pinning roots.

Changes

`dag export`

--local-only (bool): When set, only blocks present in the local blockstore are exported; missing blocks are skipped (partial CAR). User must pass --offline explicitly for a local-only DAG walk; the command does not set offline automatically.

`dag import`

--pin-roots (bool, default true): Pin optional roots listed in the CAR headers after importing.
For partial CARs, user must pass --pin-roots=false and --local-only (not both with pin-roots true).

Usage

Export a partial CAR (local blocks only; user must pass both flags):

ipfs dag export --local-only --offline <root> > partial.car

Import that partial CAR without pinning roots:

ipfs dag import --pin-roots=false partial.car
# or
ipfs dag import --local-only --pin-roots=false partial.car

References

- Export: only export blocks present locally; skip missing (partial CAR). --local-only with --offline. Support both binary and base58 link keys. - Import: support partial CARs; --local-only with -- pin-roots=false (error if both --pin-roots and --local-only set). - Fix cidFromBinString to accept base58 key format from link implementations. Signed-off-by: Chayan Das <01chayandas@gmail.com>

- remove local replace directive for go-car/v2 - upgrade to v2.16.1-0.20260306172652-7d2f4aceb070

lidel

Thank you @ChayanDass, quick first pass review below.

Also, we need to wait for go-car release with your fix before this can be merged.

Signed-off-by: Chayan Das <01chayandas@gmail.com>

ChayanDass · 2026-03-16T06:16:34Z

@lidel i’ve added the tests ,PTAL

gammazero · 2026-05-12T15:01:35Z

Triage: need to review again and check test coverage.

…ort-local-only # Conflicts: # core/commands/dag/dag.go # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum

Pass --local-only without pairing it with --offline (export) or --pin-roots=false (import); the companion is now implicit. Explicit opposites (--offline=false, --pin-roots=true) are rejected so the intent stays unambiguous. * export: imply --offline so missing blocks are not fetched over the network, which would defeat --local-only * import: imply --pin-roots=false since a partial CAR has no full DAG to pin * tests: cover the new implications and the rejected explicit-opposite combinations; drop the brittle exec.CommandContext path in favor of the existing harness

The --local-only branch now uses walker.WalkDAG with WithLocality(bs.Has) and carstorage.NewWritable, matching the MFS+unique provider in core/node/provider.go. Semantics: any input-side read error during the walk (missing block, decode failure, post-locality race) is treated as "not available locally" and the block plus its subtree are skipped. Output-side errors (writable.Put) are still surfaced. --help is updated to call out the best-effort nature. The non-local-only path is unchanged.

Pin chunker and max-file-links via a shared shallowDAGArgs so block counts are deterministic regardless of Import.* defaults or active profiles. Tighten existing assertions: * TestDagExportLocalOnly: assert exact fullCount=3 and partialCount=fullCount-1 instead of partialCount<fullCount * TestDagExportLocalOnlyImpliesOffline: assert exact partial block count, not just file Size > 0 (proves --offline was applied) Add TestDagExportLocalOnlySkipsSubtree: builds a 259-block DAG with depth>1 (256 chunks under 2 intermediates), removes an intermediate, and verifies the partial CAR is missing the intermediate plus all 174 of its descendants. Existing tests only exercised leaf removal. Extract countCARBlocks and makePartialDAG helpers used across tests.

Replace the req.Options["offline"] = true mutation with an explicit api.WithOptions(options.Api.Offline(true)) wrap after GetApi, matching the pattern already used in core/commands/dag/import.go. Clarify in comments that the walker reads from the raw blockstore (not via the kubo CoreAPI or DAGService) and therefore cannot trigger a network fetch by construction. The --offline implication exists for api.Block().Stat path resolution, not for the DAG walk itself.

ResetCids returns ctx.Err() straight from its ctx-done select, so a shutdown-during-sync surfaces as err="context canceled" while the outer ctx.Err() check at the classifier sometimes races behind the propagation and logs at Error. Classify context.Canceled the same way as keystore.ErrClosed so the message lands at Debug. Applied to both the startup and periodic classifiers. DeadlineExceeded is intentionally not included: nothing in the current call chain imposes a deadline, and a future timeout would be a real failure worth logging at Error. Closes the flake in TestProviderKeystoreSyncShutdownQuiet (10/10 local soak now green; CI hit the race 3 reruns in a row).

lidel

Thanks @ChayanDass for getting this off the ground! 🙏

I pushed a few follow-ups on top:

swapped the export walk to boxo/dag/walker so missing intermediates skip their whole subtree cleanly
--local-only now auto-implies its companion flag (--offline on export, --pin-roots=false on import) instead of asking users to pass both
tightened the tests (deterministic DAG shape, exact block counts, subtree-skip case)
small unrelated drive-by: quieted context.Canceled noise from the provider keystore sync on shutdown (fixes flaky test)

Planning to ship this in 0.42.0-rc1.

ChayanDass requested a review from a team as a code owner March 7, 2026 08:01

ChayanDass force-pushed the feat/dag-export-import-local-only branch from 2e5e224 to 9ad0c3b Compare March 7, 2026 08:03

ChayanDass and others added 2 commits March 7, 2026 13:36

Merge branch 'master' into feat/dag-export-import-local-only

fd7ee67

chore(deps): update go-car/v2 to latest master

b402796

- remove local replace directive for go-car/v2 - upgrade to v2.16.1-0.20260306172652-7d2f4aceb070

lidel requested changes Mar 8, 2026

View reviewed changes

Comment thread core/commands/dag/dag.go Outdated

Comment thread core/commands/dag/export.go Outdated

Comment thread core/commands/dag/dag.go

ChayanDass added 2 commits March 10, 2026 20:07

fix(dag): avoid CID round-trip in export and fix ci failure

7b99db6

Signed-off-by: Chayan Das <01chayandas@gmail.com>

dag: add validation and tests for --local-only flag

3b27eed

Signed-off-by: Chayan Das <01chayandas@gmail.com>

ChayanDass requested a review from lidel March 11, 2026 06:42

ChayanDass added 3 commits March 11, 2026 12:14

Merge branch 'master' into feat/dag-export-import-local-only

6a726f2

Merge branch 'master' into feat/dag-export-import-local-only

d24cde4

Merge branch 'master' into feat/dag-export-import-local-only

56a284a

ChayanDass added 2 commits March 18, 2026 22:11

Merge branch 'master' into feat/dag-export-import-local-only

6adbadf

Merge branch 'master' into feat/dag-export-import-local-only

76bc4e8

gammazero added status/blocked Unable to be worked further until needs are met need_tests labels May 12, 2026

lidel added 6 commits May 25, 2026 16:55

Merge remote-tracking branch 'origin/master' into feat/dag-export-imp…

2e088cb

…ort-local-only # Conflicts: # core/commands/dag/dag.go # docs/examples/kubo-as-a-library/go.mod # docs/examples/kubo-as-a-library/go.sum # go.mod # go.sum

chore(deps): bump go-car/v2 to latest master

51689a1

docs: changelog entry for --local-only dag export/import

6fcf6ca

lidel mentioned this pull request May 25, 2026

Release 0.42 #11227

Open

37 tasks

lidel added 2 commits May 25, 2026 21:31

lidel approved these changes May 25, 2026

View reviewed changes

lidel removed need_tests status/blocked Unable to be worked further until needs are met labels May 25, 2026

lidel merged commit 0acc688 into ipfs:master May 25, 2026
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(dag): add --local-only to dag export and import#11229

feat(dag): add --local-only to dag export and import#11229
lidel merged 18 commits into
ipfs:masterfrom
ChayanDass:feat/dag-export-import-local-only

ChayanDass commented Mar 7, 2026 •

edited by lidel

Loading

Uh oh!

lidel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChayanDass commented Mar 16, 2026

Uh oh!

gammazero commented May 12, 2026

Uh oh!

lidel left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ChayanDass commented Mar 7, 2026 • edited by lidel Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add --local-only to dag export and dag import

Summary

Changes

dag export

dag import

Usage

References

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChayanDass commented Mar 16, 2026

Uh oh!

gammazero commented May 12, 2026

Uh oh!

lidel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChayanDass commented Mar 7, 2026 •

edited by lidel

Loading

Add `--local-only` to `dag export` and `dag import`

`dag export`

`dag import`