feat(wal): add segmented write-ahead log#6
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cf17cfbdbd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| self.active_id = self.active_id.checked_add(1).ok_or(WalError::SegmentIdExhausted)?; | ||
| self.active = segment::open_segment(&self.dir, self.active_id)?; |
There was a problem hiding this comment.
If segment::open_segment fails during rotation (e.g., ENOSPC or permission error), active_id has already been incremented, so the Wal instance is left internally inconsistent: it still holds the old file handle but now reports a different segment id for future errors/LSNs and can skip ids on retry. This turns a transient I/O failure into persistent state corruption for the in-memory writer.
Useful? React with 👍 / 👎.
| self.active.write_all(&encoded).map_err(|source| { | ||
| WalError::io(segment::segment_path(&self.dir, self.active_id), source) | ||
| })?; | ||
| self.active_len = checked_add_len(self.active_len, encoded.len())?; |
There was a problem hiding this comment.
write_all may return an error after writing part of the buffer, but active_len is only advanced on success. In disk-full/intermittent I/O scenarios, a failed append can still leave a partial record on disk while in-memory offset stays stale, so later successful appends get incorrect LSNs and recovery truncates at the earlier incomplete tail, potentially discarding records written after the failure.
Useful? React with 👍 / 👎.
1 participant
Pull Request
Linear Issue
N/A
Summary
What: Adds
cloud9-wal, a small segmented write-ahead log crate with typed opaque records, explicit sync, checksummed headers/payloads, rotation, and recovery by truncating only incomplete tails.Why: Raft needs a durable storage primitive that is simpler than a database and stricter than ad hoc file writes. The WAL keeps byte durability separate from Raft, SQL, and KV semantics.
Lines added: +727
Test Plan
cargo test -p cloud9-wal -- --nocapturecargo clippy -p cloud9-wal --all-targets -- -D warningscargo test --workspace -- --nocapturecargo clippy --workspace --all-targets -- -D warningsRepro / Showcase
N/A. Tests cover append/reopen, segment rotation, incomplete-tail truncation, checksum corruption, missing segments, reserved kinds, and oversized records.
Tests Added
Documentation
N/A
Notes for Reviewers
The crate intentionally owns only byte durability. Higher layers should encode Raft hard state, log entries, truncations, and snapshots as caller-owned record payloads.