PR-06: NVS configuration and LittleFS data storage#9
Merged
Conversation
…rtifacts - Feature spec with 2026-06-11 clarifications (>=30-day retention, settable interval items, reservoir flags deferred to PR-05) - Plan with verified research D1-D10 (littlefs image build, NVS on linux target, atomicity model), data model, contracts, quickstart - 36 tasks across 8 phases; cross-artifact analysis findings fixed (FR-013 Locked* decorators, 10-metric budget cap, linux-target littlefs exclusion, sdkconfig overlay names) Spec: 003-nvs-littlefs-storage
PR-02 provides the interfaces component and the host test app this feature extends (T001, research D10). # Conflicts: # .specify/feature.json
Phase 1+2 of the storage feature: storage component skeleton with the linux-target guard for the upcoming littlefs/StorageMount code, committed littlefs seed directory, and the host test app extended for real NVS on the linux preview target (custom partition table with an nvs partition, nvs_flash REQUIRES). The Unity runner is split into per-suite run_*_tests() functions in a shared test_main.cpp; the config-store and data-storage suites are registered empty. Tasks: T002, T003, T004, T005 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
User story 1 (P1 MVP) of the storage feature: typed, validated configuration in NVS with compiled-in factory defaults. - IConfigStore interface header (host-includable, no IDF includes) with the defaults/ranges from data-model.md as the single source of truth - NvsConfigStore: namespace wscfg, one NVS entry per item, floats as u32 bit patterns, per-operation RAII handles (factoryReset erases the partition, so no handle is ever held), getters shadow missing or out-of-range stored values with defaults, setters reject out-of-range input, factory reset via nvs_flash_erase_partition + nvs_flash_init, credential values never logged - Header-only MockConfigStore holding the same contract invariants, with injectable raw stored state and write-failure simulation for later PRs - Host suite against the real linux-target NVS: defaults on erased NVS, round-trip + persistence across re-init, rejection at every documented bound, stored-value shadowing, factory reset, credential set/clear and a never-logged capture test (stdout/stderr redirected during the credential operations) Tasks: T006, T007, T008, T009, T010, T011, T012, T013 (T014 run deferred to the main session) Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Verified in the pinned espressif/idf:v6.0.1 container on the linux preview target. Docker cannot mount the OneDrive tree on this machine; verification runs from an rsync copy under /tmp. Spec: 003-nvs-littlefs-storage
Sensor history, rotating event log and storage statistics interface per contracts/IDataStorage.md. Contract-level bounds (kMaxMetrics, kEventDetailMaxLen, event categories) live here as the single source of truth for implementations, mocks and tests. Task: T015 Spec: 003-nvs-littlefs-storage
In-memory IDataStorage holding the same contract invariants as the real store (FR-012): inclusive chronological queries, distinct-metric cap, detail truncation, newest-first events, internal bounding. Rejects metric names that would be unsafe directory names (empty, '/', '..') to stay conformant with LittleFsDataStorage. Task: T016 Spec: 003-nvs-littlefs-storage
POSIX-stdio data storage skeleton with injectable base path and stats provider (no esp_littlefs/IDF includes, builds on the linux preview target). History methods stubbed for T021, event methods for T024. Registered in the storage component for both targets. Task: T017 Spec: 003-nvs-littlefs-storage
Inclusive chronological range queries, empty-result semantics (no data / unknown metric / t0>t1, FR-009), unsafe-metric-name rejection, and MockDataStorage held to the same contract (FR-012). Per-test POSIX temp dirs with RAII cleanup; written against the T017 stub, red until T021. Task: T018 Spec: 003-nvs-littlefs-storage
Chunk sealing at 1024 records, ring eviction at the 11th chunk, 30-day retention at the 5-min default interval (8640 records, no eviction, FR-010), SC-004 endurance at 10x the per-metric bound, and 11th-distinct-metric rejection. Endurance appends assert on an aggregate failure count to keep the loop fast. Task: T019 Spec: 003-nvs-littlefs-storage
A partial trailing record (size % 8 != 0, simulated power loss) is logically truncated on read with earlier records intact, and the next append repairs the tail so the chunk stays 8-byte aligned (research D5, contract invariant 2). Task: T020 Spec: 003-nvs-littlefs-storage
History per data-model.md: /hist/<metric>/<first_epoch>.dat chunks of 8-byte little-endian records, sealed at 8 KiB, ring-evicted at 10 chunks per metric, 10-distinct-metric budget guard. Durability via fflush+fsync per append; torn tails logically truncated on read and repaired (truncate to valid prefix) before the next append. Active chunk is derived from the files, so restarts need no recovery step. Event methods remain T024 stubs. Verified locally: full US2 suite (12 tests) green against a functional Unity stand-in; linux-target run deferred to T022. Task: T021 Spec: 003-nvs-littlefs-storage
Framed-record round-trip, newest-first retrieval with maxCount, truncate-and-switch rotation (oldest half dropped, newest always retained), burst within the 32 KiB budget, torn-tail marker/length detection with repair-on-append, unknown-category passthrough, >120-byte detail truncated-not-rejected, restart active-file detection, and mock event-bound conformance. Written first: the suite fails against the T024 stubs. Task: T023 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rotating two-file event log per data-model.md: /events/0.log +
1.log, 0xE7-framed records {marker, uint32 LE epoch, uint8
category, uint8 detail_len, detail}, 16 KiB per-file cap with
truncate-and-switch rotation (oldest half dropped, newest always
retained), 120-byte detail truncation, fflush+fsync per append,
torn tails skipped on read and repaired before append.
The active file is derived statelessly after restart: the file
whose last valid record carries the newest epoch (ties broken
toward the smaller file; both empty -> 0). getEvents returns
newest-first across both files, bounded by maxCount.
Task: T024
Spec: 003-nvs-littlefs-storage
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…, 0 failures) Spec: 003-nvs-littlefs-storage
Header-only mutex-serializing IConfigStore decorator (FR-013), mirroring PR-02's LockedWaterPump pattern exactly: composition over an IConfigStore reference, std::lock_guard around every interface call, host-testable pure C++ (<mutex> via pthread on both ESP-IDF and the linux preview target). Base implementations stay unsynchronized per research.md D9. Task: T026 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Header-only mutex-serializing IDataStorage decorator (FR-013), same LockedWaterPump pattern: composition over an IDataStorage reference, std::lock_guard around every interface call. Serializes the stateless-on-disk append paths (active chunk/event-file derivation, rotation) that would interleave under concurrent task access. Host-testable pure C++ per research.md D9. Task: T027 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per the mechanism PR-02's LockedWaterPump test established: every interface method exercised through the wrapper against the instrumented mock (call counts prove delegation; accepted writes, validation rejections, truncation, newest-first retrieval, stats passthrough and simulated persistence failures all behave identically behind the mutex), plus a real-store spot check each (LockedConfigStore over NvsConfigStore on linux-target NVS, LockedDataStorage over LittleFsDataStorage on a temp dir). Task: T028 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
esp_vfs_littlefs_register with partition_label="storage", base_path="/storage" and format_if_mount_failed (research D2 — legacy LittleFS.begin(true) parity: corrupted FS reformats, never bricks), plus the esp_littlefs_info stats provider that boot wiring injects into LittleFsDataStorage (FR-007/FR-008). Built only in the non-linux CMake branch (esp_littlefs has no linux port, research D4); littlefs is a PRIV_REQUIRES on the managed joltwallet__littlefs component already pinned in main/idf_component.yml — no manifest change, dependencies.lock stays valid. Task: T029 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
littlefs_create_partition_image(storage ../storage_image FLASH_IN_PROJECT) in main/CMakeLists.txt per research D1 — the canonical placement from the joltwallet/littlefs example (a component CMakeLists after idf_component_register; the relative seed path resolves against firmware/main, landing on the committed firmware/storage_image). Registers an ALL target: build/storage.bin is produced on every build and attached to idf.py flash, giving HIL a deterministic fresh-flash filesystem. Task: T030 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
NVS init with the standard ESP_ERR_NVS_NO_FREE_PAGES / NEW_VERSION_FOUND erase-and-retry recovery, StorageMount mount-or-format of /storage, and a one-line usage log (parity: storage usage in the serial status block, FR-008). Store instances are function-local statics constructed strictly after pumps_force_off() — the pumps-OFF-first invariant stays the first action in app_main — and wrapped in the Locked* decorators so every later consumer (console REPL task, controllers) goes through the mutex (FR-013). Storage failures are logged and non-fatal: config falls back to compiled-in defaults and the pump safety loop never depends on storage. ESP_LOG only, no business logic in main. Task: T031 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… tests, 0 failures) Spec: 003-nvs-littlefs-storage
Define diag_console_register_storage() and register the config and storage commands in diag_console_start(), wiring the LockedConfigStore and LockedDataStorage instances from app_main into the REPL. Completes the HIL verification path for config persistence, factory reset and data storage on the rig. Task: T032 Spec: 003-nvs-littlefs-storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- CI verify-binaries step asserts build/storage.bin exists (T033) - parity-checklist §6: deliberate divergences for the ESP-IDF storage port — NVS config, empty-SSID unconfigured state, bounded history, new event log, interface split, deferred reservoir flags (T034) - firmware/CLAUDE.md: storage component, layout, console commands (T035) Spec: 003-nvs-littlefs-storage
rev1_devkit + rev2 both produce wateringsystem.bin and the 960 KiB build/storage.bin (FLASH_IN_PROJECT); host suite green (44 tests). All PR-06 tasks complete. Spec: 003-nvs-littlefs-storage
…gaps, contract docs
- F1: storeEvent no longer appends when stat() fails on an event file that
holds valid bytes (would corrupt the frame boundary while returning true);
an absent not-yet-created file still appends as before.
- F2: add getStorageStats() provider-path test (no provider -> {0,0}; fake
provider -> injected total/used).
- F3: add non-monotonic-epoch chunk-bump test (sealed chunk + earlier-epoch
append: no name collision, earlier record retrievable).
- D1: document that NvsConfigStore.factoryReset() erases the entire default
NVS partition, not just the wscfg namespace.
- D2: note getEvents() newest-first ordering assumes monotonic epochs.
- D3: note LockedConfigStore/LockedDataStorage give per-call atomicity only.
Spec: 003-nvs-littlefs-storage
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR-06: NVS configuration + LittleFS data storage (Phase 2)
Ports the storage layer to ESP-IDF as two redesigned, host-includable contracts.
Spec/plan/contracts:
specs/003-nvs-littlefs-storage/. Implements FR-013 (NVSconfig + factory defaults); enables FR8 (littlefs assets) and the history part of
FR7. No migration of Arduino-era data (clean start).
What's in it
IConfigStore→NvsConfigStore: typed config in thenvspartition(namespace
wscfg, one entry per item), compiled-in factory defaults applied onmissing/erased/out-of-range entries, explicit factory reset. Replaces the legacy
string-keyed JSON blob.
IDataStorage→LittleFsDataStorage: bounded sensor history (per-metric8-byte-record chunk files, 8 KiB chunks, 10-chunk ring eviction, ≥30-day
retention, 10-metric cap) and a rotating event log (two 16 KiB files, newest
always retained), plus filesystem usage stats. POSIX stdio with an injectable
base path — runs on the host; only
StorageMount(mount-or-format, usage) isesp32-only and excluded from the linux build.
LockedConfigStore/LockedDataStorage: mutex decorators for cross-taskaccess (FR-013), mirroring the PR-02
LockedWaterPumppattern.app_main(pumps-OFF-first invariant preserved; all storageinit non-fatal) and
config/storagediagnostic-console commands for HIL.flash (
build/storage.bin), verified in CI.Decisions (clarified 2026-06-11)
≥30-day history retention;
sensorReadInterval/dataLogIntervalare settable;reservoir flags deferred to PR-05. Deliberate divergences from the Arduino storage
layer are recorded in
docs/parity-checklist.md§6.Verification
storage.bin(960 KiB)produced on each.
POSIX file layer).
type-design-analyzer (CP3). One real bug fixed (
storeEventtorn-tailstat-failure), two test gaps closed, three contract-doc clarifications added.
is non-fatal and never touches the pump loop.
Remaining for the bench (not blocking merge)
HIL checklist in
specs/003-nvs-littlefs-storage/quickstart.md— configpersistence across power cycle, factory reset, corruption recovery — run on the
rev1 rig.
🤖 Generated with Claude Code