Skip to content

CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork) #102

Open
Open
CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork)#102

Description

@bourgeoa
bourgeoa
opened on Aug 31, 2026

@jeswr
This may help to join effort
The overlay do not appear so high

CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork)

Analysis for cross-branch comparison, prepared to be shared with @jeswr.
Date: 2026-08-31
Scope: the recurring lock-timeout crashes + sustained 100 % CPU observed on a
Community Solid Server (CSS) with a large account count, and how two
independent branches addressed it.

1. Background / problem statement

A CSS v7.2.0 deployment (pivot-test.solidproject.org:3200, ~1000 accounts) exhibited:

  • Recurring crashes (unhandledRejection: Lock expired after 30000ms .../.internal/accounts/data/<uuid>), roughly every 30 minutes, matching the account-cleanup timer.
  • Sustained 95–100 % CPU with zero traffic, from ~10 minutes after boot.
  • Sporadic proper-lockfile race crash (Cannot read properties of undefined (reading 'updateTimeout')) under CPU starvation.

Root cause chain (validated)

  1. Full-tree sweep walks. Every expiring-storage sweep
    (WrappedExpiringStorage.removeExpiredEntries) calls entries() on a
    ContainerPathStorage, which delegates to the underlying
    JsonResourceStorage.entries() and walks the entire /.internal/ tree
    (recursively reading every account data/index/idp file, thousands of files),
    then filters by prefix. The 4 sweeps (cookies / forgot-password / OIDC
    adapter / tokens, at 10/15/20/25 min) overlap → sustained CPU saturation.
    This is a pre-existing upstream CSS inefficiency (shipped configs use
    ContainerPathStorage over a whole-tree KeyValueStorage).
  2. Unbounded sweep deletes. The sweep used Promise.all(expired.map(delete))
    — every delete goes through the locked pipeline → lock/EMFILE/memory spikes.
  3. Fire-and-forget cleanup crash. BaseLoginAccountStorage.createAccountTimeout
    (default 30 min) is an unhandled setTimeout without try/catch; its
    storage.delete() lock-timeout rejection crashes the process.
  4. proper-lockfile 4.1.2 race (unmaintained): renewal timer already fired →
    in-flight fs callback recurses updateLock after delete locks[file] →
    lock.updateTimeout on undefined → uncaughtException.

2. The two branches

Ours jeswr's
Repo CommunitySolidServer branch clean-lock + Pivot port branch fix/clear-lock jeswr/CommunitySolidServer branch codex/solidcommunity-recovery-alpha-1 + Pivot deployment branch codex/solidcommunity-durable-deployment
CSS used published @solid/community-server 7.2.0 (unmodified) @jeswr/community-solid-server 7.1.10-alpha.2 (npm alias → fork)
Authors bourgeoa jeswr (+ Claude / Copilot co-authors)
Basis upstream 7.2.0 upstream ~7.1.9 (Mar 2026)
Commits CSS: 8 · Pivot: 2 fork: ~35 recovery commits (Aug 24–31) · Pivot: 4

The two branches are completely disjoint — no shared commits:

> 3687fb0  chore: lock corrected solidcommunity alpha releases   (codex)
> 748bdec  chore: pin alpha cache release
> d823320  docs: record first-deployment rollback command
> 259735d  feat: add durable solidcommunity.net deployment
< d5e6428  fix: move file-locker override out of shared config    (fix/clear-lock)
< 4c46845  fix: port CSS lock/sweep hardening

3. jeswr's fork — changes vs latest CSS (analysis)

Compare CommunitySolidServer/main...jeswr:.../codex/solidcommunity-recovery-alpha-1:
58 commits / 84 files, of which ~35 are the Aug 24–31 "solidcommunity
recovery" work. It also contains bourgeoa's own fa4f3fc (pod quota in
subdomain mode, PR #2210) and b48edf8 (Static Asset root expansion).

3.1 Locking ("improved clear lock")

Commit Change
a6dc692 maxHoldDuration on WrappedExpiringReadWriteLocker — an absolute cap on total lock hold time, counted from acquisition and never renewed by activity. Default 0 = unlimited (backward compatible, stock configs unchanged). Opt-in config/util/resource-locker/file-capped.json sets 1 hour. Purpose: a stream of trickle-readers can no longer renew a read lock forever and starve a waiting writer.
a4f44e9 Keep write locks alive while request data flows — LockingResourceStore.lockedRepresentationWrite monkey-patches the incoming representation's data.read to call maintainLock() on every chunk (slow uploads no longer drop the lock). If the lock expires mid-write: restore the hook + data.destroy(error) so the source store can't keep writing unprotected.
0119654 Restore the stream read method after lock release — preserve the exact function reference (no .bind() wrappers accumulating), restore it on unlock.
ded8013, 03c6dc8 Remove unsafe stream cast; PR-review finding fix.
Redis 2703fe3 renewed TTL (a crashed holder can't deadlock peers), 7c008e6 don't wipe peer instances' locks on boot, ca24f8c wipe namespace only on single-instance shutdown.
Cluster 117f21b back off + budget cluster worker restarts.

3.2 Expiring sweeps

Commit Change
02019c1 WrappedExpiringStorage now implements Finalizable (finalize() clears the interval, unref() kept) and gains an optional jitter param (default 0.15 × timeout) so the 4 sweep instances don't all fire at the same instant. All four are wired into urn:solid-server:default:Finalizer.
fbea7e6 feat: sweep expired notification channels from storage.

3.3 Container listing streaming (biggest perf change)

Commit Change
9b214f0 + follow-ups (64c849b, f40d969, 3519acf, 1debc2a, 1824600, cc92f14, a4cd01d, 689427f, 680b85c, 65c9892) DataAccessorBasedStore.getRepresentation emits container listings as a lazy Readable<Quad> (streamContainerRepresentation) instead of folding every child into one N3 store. The ldp:contains list now lives in the streamed body (one marker kept in metadata so AllowAcceptHeaderWriter's empty-container check still works); JsonResourceStorage + SingleContainerJsonStorage read members from the body (draining also releases the container read lock before recursing). Measured: a 15k-child container under 20 concurrent reads drops peak live heap from ~3056 MB to ~140 MB (flat).

3.4 Caching

Commit Change
842d943 Reusable promise-cache abstraction.
86603d4 Cache the representation-converter negotiation result.
3a6278b Read each effective ACL only once per request.
3e31908 Cache ACL documents with write-driven invalidation.
2e21de4, 74adac3 Parse/serialize buffered quads synchronously in QuadUtil/readableToQuads.
86b8eff Stream file metadata quads (later reverted in 4c7632d).

3.5 Identity

Commit Change
ddb2064 Reuse known account ID on cookie refresh (drops a redundant read).
fcc8e93 Dedicated OIDC client storage — ExpiringAdapterFactory takes a second clientStorage; the Client model uses it, everything else uses the main storage (falls back if unset). Exposes urn:solid-server:default:IdpAdapterExpiringStorage (transient provider state) and urn:solid-server:default:IdpClientAdapterExpiringStorage (persistent dynamic registrations) as named components.

⚠️ These two @ids exist only in the fork — not in published CSS 7.2.0.
The Pivot solidcommunity.net overrides use them: adapter → MemoryMapStorage,
clients → persisted WrappedExpiringStorage(ContainerPathStorage(/idp/adapter/)).

3.6 Mapping + misc

Commit Change
dee7e9f ExtensionBasedMapper "learns" extension lookup order: sample directory (≤128 entries via opendir), rank the ≤24 most frequent $.ext extensions, LRU-cache 1024 directory profiles, dedupe concurrent scans; stat exact filename first, probe extensions by frequency, full readdir fallback + profile refresh.
4fc6428 Skip the log-format pipeline for filtered-out log levels.
5cec047 Skip stack-trace capture for client-error control flow.

3.7 Packaging

0a61514 release prep · 91d840e package production hotfixes · c892276
canonicalize fork component metadata · 1ea23d4 support installation through
CSS npm alias.

4. Our branch — changes vs latest CSS

CSS clean-lock (8 commits)

Commit Change
53aa83297 fix(account): do not crash on lock timeout during account cleanup — try/catch in BaseLoginAccountStorage.createAccountTimeout.
9bb140418 fix(storage): bound delete concurrency in expiring-storage sweep (batch 32 in WrappedExpiringStorage).
94edf2759 feat(storage): configurable sweep batchSize (optional).
a807630b7 fix(storage): destroy representation stream when a read lock expires mid-read (LockingResourceStore, timedOut flag).
81c8027ed fix(locking): bound lock-acquisition retries — FileSystemResourceLocker retryCount: -1 → 600 (~30 s max wait).
afb8cb181 feat(config): give the IDP expiring adapter storage @id urn:solid-server:default:ExpiringAdapterStorage.
9cb8dd07a fix(config): scope expiring-storage sweeps to their own containers — 4 config files.
fc598140b fix(deps): guard proper-lockfile updateLock race via patch-package.

The core root-cause fix (9cb8dd07a): drop ContainerPathStorage and point
JsonResourceStorage.container at the subpath:

MaxKeyLengthStorage(JsonResourceStorage(
  source:    ResourceStore_Backend,
  baseUrl:   variable:baseUrl,
  container: "/.internal/<subpath>/"
))
  • cookies → /.internal/accounts/cookies/
  • forgot-password → /.internal/accounts/forgot-password/
  • tokens → /.internal/idp/tokens/
  • adapter → /.internal/idp/adapter/

On-disk paths are unchanged → backward compatible, no migration.
Validated on the box: node CPU 0.2–0.7 %, load ~0.00, flat pings.

Pivot fix/clear-lock (2 commits)

Commit Change
4c46845 Port: proper-lockfile patch + CSS adapter-@id patch, postinstall, config/pivot-overrides.json (4 scoped sweeps incl. adapter), SafeBaseLoginAccountStorage (+ src/index.ts export).
d5e6428 Move the file-locker override out of the shared config → config/pivot-file-locker-overrides.json (imported by prod/suffix/dev configs, not test.json).

Pivot-specific constraints solved along the way:

  • TS2415 private logger collision → safeLogger.
  • componentsjs-generator can't resolve IndexedStorage generics from published
    CSS d.ts → storage: any, non-generic class.
  • nested config params flattened in components.js (attemptSettings_retryCount, …).
  • test.json uses a memory locker → "Missing type for override target" → override
    moved to its own file.

5. Direct comparison

Both branches were built for the same symptom set but attack it from
opposite sides:

Concern Ours (fix/clear-lock) jeswr (codex/...)
Sweep cost (root cause) Scope each sweep to its own container — eliminates the full-tree walk Jitter sweeps + move cookies/adapter to memory in deployment
Lock hold time keep 30 s idle expiry maxHoldDuration absolute cap (1 h opt-in)
Lock acquisition Bounded retries (retryCount: 600) — (memory LockStorage, single-process)
Crash guard proper-lockfile race guard + SafeBaseLoginAccountStorage try/catch worker-restart back-off/budget
Write-path lock renewal — (only read-side stream destroy, a807630b7) keep write locks alive during data flow + destroy stream on expiry
Peak memory (listings) — streamed container listings (O(children) → O(1))
Caching — promise/ACL/converter caches + @jeswr/css-cached-storage
Runs on unmodified published CSS 7.2.0 requires the fork (adapter @ids + maxHoldDuration are fork-only)

Key divergence on the sweep root cause

The fork does not change the sweep's storage architecture. Its on-disk
IdpClientAdapterExpiringStorage is still
WrappedExpiringStorage(ContainerPathStorage(/idp/adapter/, KeyValueStorage)),
so the sweep still walks all of /.internal/ — the exact O(n) full-tree walk
that caused the sustained CPU in our deployment. jeswr's countermeasures are:
jitter (spread the cost), memory-backed hot storages (avoid disk sweeps for
cookies/adapter), plus the caching layer.

Measured numbers

before after
CPU at 1000 accounts 95–100 % sustained 0.2–0.7 % (ours)
Response time volatile 1000–4000 ms flat ~0 ms (ours)
Container listing (15k children, 20 concurrent reads) ~3056 MB peak heap ~140 MB (jeswr)

6. Compatibility & synergy

The two approaches are complementary, not competing:

  1. Our scoped sweeps would benefit the fork too — it would eliminate the
    remaining on-disk client-storage sweep walk (currently jittered but still O(n)).
  2. jeswr's maxHoldDuration + write-lock renewal would strengthen our branch
    (we only bounded acquisition retries; his work bounds hold time and protects
    slow uploads).
  3. Both share a clear principle: sweeps and lock renewals must be bounded —
    ours bounds work (scope + batch + retries), jeswr's bounds time
    (maxHoldDuration) and memory (streaming + caching).

Hard incompatibility: the codex Pivot branch cannot run on published CSS
without the fork (fork-only @ids IdpAdapterExpiringStorage /
IdpClientAdapterExpiringStorage, and maxHoldDuration). Our branch runs on
unmodified published CSS 7.2.0. Any future merge of the fork's work into
upstream would unblock the codex branch.

7. Suggested next steps (for discussion)

  • Decide whether to upstream the fork's "solidcommunity recovery" work
    (maxHoldDuration, write-lock renewal, streamed container listings,
    promise/ACL caches) as PRs against CSS.
  • Consider porting our scoped-sweep config change onto the fork, so the
    remaining on-disk sweep is O(own container) instead of O(whole tree).
  • Evaluate maxHoldDuration + write-lock renewal for our clean-lock
    branch (they address a real starvation scenario we did not cover).
  • Cross-check the adapter-@id naming (ExpiringAdapterStorage ours vs
    IdpAdapterExpiringStorage/IdpClientAdapterExpiringStorage fork) for a
    single canonical convention.

Appendix — useful references

CSS lock/sweep hardening: fix/clear-lock vs codex/solidcommunity-durable-deployment (jeswr CSS fork)

Analysis for cross-branch comparison, prepared to be shared with @jeswr.
Date: 2026-08-31
Scope: the recurring lock-timeout crashes + sustained 100 % CPU observed on a
Community Solid Server (CSS) with a large account count, and how two
independent branches addressed it.

1. Background / problem statement

A CSS v7.2.0 deployment (pivot-test.solidproject.org:3200, ~1000 accounts) exhibited:

  • Recurring crashes (unhandledRejection: Lock expired after 30000ms .../.internal/accounts/data/<uuid>), roughly every 30 minutes, matching the account-cleanup timer.
  • Sustained 95–100 % CPU with zero traffic, from ~10 minutes after boot.
  • Sporadic proper-lockfile race crash (Cannot read properties of undefined (reading 'updateTimeout')) under CPU starvation.

Root cause chain (validated)

  1. Full-tree sweep walks. Every expiring-storage sweep
    (WrappedExpiringStorage.removeExpiredEntries) calls entries() on a
    ContainerPathStorage, which delegates to the underlying
    JsonResourceStorage.entries() and walks the entire /.internal/ tree
    (recursively reading every account data/index/idp file, thousands of files),
    then filters by prefix. The 4 sweeps (cookies / forgot-password / OIDC
    adapter / tokens, at 10/15/20/25 min) overlap → sustained CPU saturation.
    This is a pre-existing upstream CSS inefficiency (shipped configs use
    ContainerPathStorage over a whole-tree KeyValueStorage).
  2. Unbounded sweep deletes. The sweep used Promise.all(expired.map(delete))
    — every delete goes through the locked pipeline → lock/EMFILE/memory spikes.
  3. Fire-and-forget cleanup crash. BaseLoginAccountStorage.createAccountTimeout
    (default 30 min) is an unhandled setTimeout without try/catch; its
    storage.delete() lock-timeout rejection crashes the process.
  4. proper-lockfile 4.1.2 race (unmaintained): renewal timer already fired →
    in-flight fs callback recurses updateLock after delete locks[file] →
    lock.updateTimeout on undefined → uncaughtException.

2. The two branches

Ours jeswr's
Repo CommunitySolidServer branch clear-lock (PR #2235) + Pivot port branch fix/clear-lock (PR #144) jeswr/CommunitySolidServer branch codex/solidcommunity-recovery-alpha-1 + Pivot deployment branch codex/solidcommunity-durable-deployment
CSS used published @solid/community-server 7.2.0 (unmodified) @jeswr/community-solid-server 7.1.10-alpha.2 (npm alias → fork)
Authors bourgeoa jeswr (+ Claude / Copilot co-authors)
Basis upstream 7.2.0 upstream ~7.1.9 (Mar 2026)
Commits CSS: 8 · Pivot: 2 fork: ~35 recovery commits (Aug 24–31) · Pivot: 4

The two branches are completely disjoint — no shared commits:

> 3687fb0  chore: lock corrected solidcommunity alpha releases   (codex)
> 748bdec  chore: pin alpha cache release
> d823320  docs: record first-deployment rollback command
> 259735d  feat: add durable solidcommunity.net deployment
< d5e6428  fix: move file-locker override out of shared config    (fix/clear-lock)
< 4c46845  fix: port CSS lock/sweep hardening

3. jeswr's fork — changes vs latest CSS (analysis)

Compare CommunitySolidServer/main...jeswr:.../codex/solidcommunity-recovery-alpha-1:
58 commits / 84 files, of which ~35 are the Aug 24–31 "solidcommunity
recovery" work. It also contains bourgeoa's own fa4f3fc (pod quota in
subdomain mode, PR #2210) and b48edf8 (Static Asset root expansion).

3.1 Locking ("improved clear lock")

Commit Change
a6dc692 maxHoldDuration on WrappedExpiringReadWriteLocker — an absolute cap on total lock hold time, counted from acquisition and never renewed by activity. Default 0 = unlimited (backward compatible, stock configs unchanged). Opt-in config/util/resource-locker/file-capped.json sets 1 hour. Purpose: a stream of trickle-readers can no longer renew a read lock forever and starve a waiting writer.
a4f44e9 Keep write locks alive while request data flows — LockingResourceStore.lockedRepresentationWrite monkey-patches the incoming representation's data.read to call maintainLock() on every chunk (slow uploads no longer drop the lock). If the lock expires mid-write: restore the hook + data.destroy(error) so the source store can't keep writing unprotected.
0119654 Restore the stream read method after lock release — preserve the exact function reference (no .bind() wrappers accumulating), restore it on unlock.
ded8013, 03c6dc8 Remove unsafe stream cast; PR-review finding fix.
Redis 2703fe3 renewed TTL (a crashed holder can't deadlock peers), 7c008e6 don't wipe peer instances' locks on boot, ca24f8c wipe namespace only on single-instance shutdown.
Cluster 117f21b back off + budget cluster worker restarts.

3.2 Expiring sweeps

Commit Change
02019c1 WrappedExpiringStorage now implements Finalizable (finalize() clears the interval, unref() kept) and gains an optional jitter param (default 0.15 × timeout) so the 4 sweep instances don't all fire at the same instant. All four are wired into urn:solid-server:default:Finalizer.
fbea7e6 feat: sweep expired notification channels from storage.

3.3 Container listing streaming (biggest perf change)

Commit Change
9b214f0 + follow-ups (64c849b, f40d969, 3519acf, 1debc2a, 1824600, cc92f14, a4cd01d, 689427f, 680b85c, 65c9892) DataAccessorBasedStore.getRepresentation emits container listings as a lazy Readable<Quad> (streamContainerRepresentation) instead of folding every child into one N3 store. The ldp:contains list now lives in the streamed body (one marker kept in metadata so AllowAcceptHeaderWriter's empty-container check still works); JsonResourceStorage + SingleContainerJsonStorage read members from the body (draining also releases the container read lock before recursing). Measured: a 15k-child container under 20 concurrent reads drops peak live heap from ~3056 MB to ~140 MB (flat).

3.4 Caching

Commit Change
842d943 Reusable promise-cache abstraction.
86603d4 Cache the representation-converter negotiation result.
3a6278b Read each effective ACL only once per request.
3e31908 Cache ACL documents with write-driven invalidation.
2e21de4, 74adac3 Parse/serialize buffered quads synchronously in QuadUtil/readableToQuads.
86b8eff Stream file metadata quads (later reverted in 4c7632d).

3.5 Identity

Commit Change
ddb2064 Reuse known account ID on cookie refresh (drops a redundant read).
fcc8e93 Dedicated OIDC client storage — ExpiringAdapterFactory takes a second clientStorage; the Client model uses it, everything else uses the main storage (falls back if unset). Exposes urn:solid-server:default:IdpAdapterExpiringStorage (transient provider state) and urn:solid-server:default:IdpClientAdapterExpiringStorage (persistent dynamic registrations) as named components.

⚠️ These two @ids exist only in the fork — not in published CSS 7.2.0.
The Pivot solidcommunity.net overrides use them: adapter → MemoryMapStorage,
clients → persisted WrappedExpiringStorage(ContainerPathStorage(/idp/adapter/)).

3.6 Mapping + misc

Commit Change
dee7e9f ExtensionBasedMapper "learns" extension lookup order: sample directory (≤128 entries via opendir), rank the ≤24 most frequent $.ext extensions, LRU-cache 1024 directory profiles, dedupe concurrent scans; stat exact filename first, probe extensions by frequency, full readdir fallback + profile refresh.
4fc6428 Skip the log-format pipeline for filtered-out log levels.
5cec047 Skip stack-trace capture for client-error control flow.

3.7 Packaging

0a61514 release prep · 91d840e package production hotfixes · c892276
canonicalize fork component metadata · 1ea23d4 support installation through
CSS npm alias.

4. Our branch — changes vs latest CSS

CSS clear-lock (8 commits) — PR #2235

Commit Change
53aa83297 fix(account): do not crash on lock timeout during account cleanup — try/catch in BaseLoginAccountStorage.createAccountTimeout.
9bb140418 fix(storage): bound delete concurrency in expiring-storage sweep (batch 32 in WrappedExpiringStorage).
94edf2759 feat(storage): configurable sweep batchSize (optional).
a807630b7 fix(storage): destroy representation stream when a read lock expires mid-read (LockingResourceStore, timedOut flag).
81c8027ed fix(locking): bound lock-acquisition retries — FileSystemResourceLocker retryCount: -1 → 600 (~30 s max wait).
afb8cb181 feat(config): give the IDP expiring adapter storage @id urn:solid-server:default:ExpiringAdapterStorage.
9cb8dd07a fix(config): scope expiring-storage sweeps to their own containers — 4 config files.
fc598140b fix(deps): guard proper-lockfile updateLock race via patch-package.

The core root-cause fix (9cb8dd07a): drop ContainerPathStorage and point
JsonResourceStorage.container at the subpath:

MaxKeyLengthStorage(JsonResourceStorage(
  source:    ResourceStore_Backend,
  baseUrl:   variable:baseUrl,
  container: "/.internal/<subpath>/"
))
  • cookies → /.internal/accounts/cookies/
  • forgot-password → /.internal/accounts/forgot-password/
  • tokens → /.internal/idp/tokens/
  • adapter → /.internal/idp/adapter/

On-disk paths are unchanged → backward compatible, no migration.
Validated on the box: node CPU 0.2–0.7 %, load ~0.00, flat pings.

Pivot fix/clear-lock (2 commits) — PR #144

Commit Change
4c46845 Port: proper-lockfile patch + CSS adapter-@id patch, postinstall, config/pivot-overrides.json (4 scoped sweeps incl. adapter), SafeBaseLoginAccountStorage (+ src/index.ts export).
d5e6428 Move the file-locker override out of the shared config → config/pivot-file-locker-overrides.json (imported by prod/suffix/dev configs, not test.json).

Pivot-specific constraints solved along the way:

  • TS2415 private logger collision → safeLogger.
  • componentsjs-generator can't resolve IndexedStorage generics from published
    CSS d.ts → storage: any, non-generic class.
  • nested config params flattened in components.js (attemptSettings_retryCount, …).
  • test.json uses a memory locker → "Missing type for override target" → override
    moved to its own file.

5. Direct comparison

Both branches were built for the same symptom set but attack it from
opposite sides:

Concern Ours (fix/clear-lock) jeswr (codex/...)
Sweep cost (root cause) Scope each sweep to its own container — eliminates the full-tree walk Jitter sweeps + move cookies/adapter to memory in deployment
Lock hold time keep 30 s idle expiry maxHoldDuration absolute cap (1 h opt-in)
Lock acquisition Bounded retries (retryCount: 600) — (memory LockStorage, single-process)
Crash guard proper-lockfile race guard + SafeBaseLoginAccountStorage try/catch worker-restart back-off/budget
Write-path lock renewal — (only read-side stream destroy, a807630b7) keep write locks alive during data flow + destroy stream on expiry
Peak memory (listings) — streamed container listings (O(children) → O(1))
Caching — promise/ACL/converter caches + @jeswr/css-cached-storage
Runs on unmodified published CSS 7.2.0 requires the fork (adapter @ids + maxHoldDuration are fork-only)

Key divergence on the sweep root cause

The fork does not change the sweep's storage architecture. Its on-disk
IdpClientAdapterExpiringStorage is still
WrappedExpiringStorage(ContainerPathStorage(/idp/adapter/, KeyValueStorage)),
so the sweep still walks all of /.internal/ — the exact O(n) full-tree walk
that caused the sustained CPU in our deployment. jeswr's countermeasures are:
jitter (spread the cost), memory-backed hot storages (avoid disk sweeps for
cookies/adapter), plus the caching layer.

Measured numbers

before after
CPU at 1000 accounts 95–100 % sustained 0.2–0.7 % (ours)
Response time volatile 1000–4000 ms flat ~0 ms (ours)
Container listing (15k children, 20 concurrent reads) ~3056 MB peak heap ~140 MB (jeswr)

6. Compatibility & synergy

The two approaches are complementary, not competing:

  1. Our scoped sweeps would benefit the fork too — it would eliminate the
    remaining on-disk client-storage sweep walk (currently jittered but still O(n)).
  2. jeswr's maxHoldDuration + write-lock renewal would strengthen our branch
    (we only bounded acquisition retries; his work bounds hold time and protects
    slow uploads).
  3. Both share a clear principle: sweeps and lock renewals must be bounded —
    ours bounds work (scope + batch + retries), jeswr's bounds time
    (maxHoldDuration) and memory (streaming + caching).

Hard incompatibility: the codex Pivot branch cannot run on published CSS
without the fork (fork-only @ids IdpAdapterExpiringStorage /
IdpClientAdapterExpiringStorage, and maxHoldDuration). Our branch runs on
unmodified published CSS 7.2.0. Any future merge of the fork's work into
upstream would unblock the codex branch.

7. Suggested next steps (for discussion)

  • Decide whether to upstream the fork's "solidcommunity recovery" work
    (maxHoldDuration, write-lock renewal, streamed container listings,
    promise/ACL caches) as PRs against CSS.
  • Consider porting our scoped-sweep config change onto the fork, so the
    remaining on-disk sweep is O(own container) instead of O(whole tree).
  • Evaluate maxHoldDuration + write-lock renewal for our clear-lock
    branch (they address a real starvation scenario we did not cover).
  • Cross-check the adapter-@id naming (ExpiringAdapterStorage ours vs
    IdpAdapterExpiringStorage/IdpClientAdapterExpiringStorage fork) for a
    single canonical convention.

Appendix — useful references

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions