feat: atrium-repo (#272)

* chore: Create empty `atrium-repo` library crate

* repo: Logic for parsing and traversing MST nodes

* repo: Indexed reader for CAR files

* repo: APIs to fetch keys, collections, and records from a repository

* Significant refactor; implement basic MST update logic

* Full MST implementation

* More refactoring :)

* Use `traverse` for `split_subtree`

* Fix a few lingering bugs in node deletion

* Update firehose example

* More validation in mst dual-layer deletion test

* `Tree::get_path`

* Only verify SHA2_256 CAR blocks

* Documentation pass

* MST: Handle edge case where we need to insert new tree entry at leftmost position in parent

* Fix repo tests

* More docs

* Simplify `Tree::keys`

* Return keys from `Tree::keys` in lexicographic order

* Create a high-level `Commit` helper struct

* Add new `Tree::entries` function to return key/value tuples

* Add documentation clarifying that tree iteration does not work against firehose records

* Whoops :)

* Expose hashing algorithm to `AsyncBlockStoreWrite::write_block`

* Forgot to commit this import :(

* Gracefully handle multi-writes to `CarStore`

* Rename `Repository::new` to `open`

* Outline `algos::compute_depth`

* Correct the docs for `Node::find_ge`

* Panic if we attempt to serialize a `Node` with two adjacent tree entries

* Misc. improvements

* More `Tree` documentation

* Remove `AsyncBlockStoreWrite::delete_block`

Why: This function is a footgun that will likely be misused.
Deleting a block is not typically safe because it could have multiple incoming references (and thus deletion converts those into dangling references).
The only way to safely delete a block would be to wholistically enumerate an archive file and collect a list of all referenced blocks.

* Fix a bug when writing out a new CAR block

* API updates; Implement `CarStore::create`

* Update firehose example to remove CID compat

* Move all algorithms to `algos` module

* Fix warnings

* clippy + fmt

* Full CRUD functionality for `Repository`

* Add a basic differencing blockstore layer

* Remove `async-trait` dep

* Add my name to `atrium-repo` authors

* Add `Commit::rev`

* Add github workflow

* Remove vscode from gitignore

* Use `RecordKey` in `Repository::get` API

* Add `extract_path` APIs

* Fix some lingering bugs in `CarStore`

* Add `_raw` API variants to `Repository`

* Add another 2 block test for CarStore

* Add an `Error::Other` variant for block storage

* Add a test for `set_root`

* Fix one more bug encountered when writing blocks

* `extract_path` -> `extract`; add `_into` variants

* Downgrade cid reference to straight copy instead

* Deduplicate some code

* Add extraction test

* Builder `sign` -> `finalize`

* Slightly expand extraction test for exclusion proofs

* Remove the hacked-together `CarStore::set_root`

* Add complex extraction test

* Fix a bug causing extraction to fail to extract the MST root

* Add `MemoryBlockStore::contains`

* clippy fix

* Add `Repository::get_raw_cid` API

* Return both a `CommitBuilder` and `Cid` from add/update APIs

* `Repository::tree` API

* Perform deserialization in `get_raw` APIs

* Add prefixed iteration APIs to `Tree`

* Use `serde-bytes` to assist with deserializing byte arrays

* Fix warnings

* Add `export` APIs

* Fix test

* Remove old `commit` tests

* Remove `CommitBuilder` example doc comment

* Fix a potential panic; cleanup

* Apply suggestions from code review

* `cargo clippy`

* Apply fixes suggested by clippy

* Fix signing

* Re-export ipld's multihash type.

---------

Co-authored-by: Jack Grigg <[email protected]>

Author: DrChat

.github/workflows/repo.yml

@@ -0,0 +1,22 @@
+name: Repo
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    branches: ["main"]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Build
+        run: |
+          cargo build -p atrium-repo --verbose
+      - name: Run tests
+        run: |
+          cargo test -p atrium-repo