Skip to content

Releases: SciQLop/Sciqlop-cache

v0.2.1

Choose a tag to compare

@jeandet jeandet released this 23 Sep 16:45

First release since 0.1.16 on PyPI. It includes the 0.2.0 work (tagged, never published) and the fixes below.

Highlights

  • Free-threaded Python (3.13t / 3.14t) is supported and tested. The extension runs without the GIL, and a new multi-threaded torture test runs in CI on 3.14t.
  • Fixed a GIL deadlock on regular Python. A thread inside transact() (or iterating) plus another thread calling iterkeys, incr, get_meta, check, clear, size, evict, stats, ... could freeze the interpreter. Every binding that takes the store lock now releases the GIL.
  • Fixed data loss on rollback. delete, pop or evict_tag of a large (file-backed) value inside a transact() that rolled back lost the value. Value files are now queued for deletion in the same transaction as the row change, so a rollback keeps them.
  • Fixed orphan files on Windows. A file still mapped by a live buffer could not be deleted and was left behind. It is now retried in the background until it can be removed.
  • Fixed a crash: dropping a FanoutCache / FanoutIndex while iterating over its keys segfaulted.

diskcache compatibility (from 0.2.0)

  • Non-str keys, retry= on every method, get(tag=True, expire_time=True), diskcache constructor kwargs (directory=, size_limit=, shards=, ...).
  • Index is a MutableMapping; cache[missing] / Index.pop(missing) raise KeyError; Timeout exception.
  • touch() matches diskcache: optional expire, no resurrection of expired keys, False on a miss, int expire accepted (#14).

Performance

Measured against the previous commit, same machine, interleaved runs:

  • Deleting large values is faster: del -28%, pop -19%, evict_tag -84% per row. Files are now removed by the background thread, so their disk space comes back about 5 s later.
  • set -5%, batched set -11%, get unchanged.
  • The README benchmarks are refreshed, and the Performance section now documents the test setup.

Build

  • nanobind 2.15, SQLite 3.53.4, Catch2 3.16, robin-map 1.4.1.
  • cpp_utils is pinned to a fixed commit instead of tracking main.

Full changes: v0.1.16...v0.2.1

v0.1.16

Choose a tag to compare

@jeandet jeandet released this 14 Sep 20:59

v0.1.15 fixed close() leaking mmap handles and the main sqlite3 connection on Windows (issue #13), but introduced a heap-use-after-free in Database::close() in the process: sqlite3_close_v2() now fully frees the connection once BEGIN/COMMIT are finalized, and close() then called db.reset(), whose deleter called sqlite3_close() again on that freed pointer. That's what crashed test_wheels(windows-latest, 3.14t) in CI ("Windows fatal exception: access violation").

v0.1.15 never reached PyPI -- its release workflow failed before the upload step, so nothing broken was published.

Fixed here (db.release() instead of db.reset()) and verified deterministically with AddressSanitizer: the bug reproduced on the very first close() call before the fix and is clean after it, across the full C++ suite and all 146 Python interface tests.

v0.1.15

Choose a tag to compare

@jeandet jeandet released this 14 Sep 20:08

Follow-up to v0.1.14 for issue #13. The previous fix only released the mmap LRU cache in close(); the main sciqlop-cache.db connection itself still leaked on Windows because Database's own BEGIN/COMMIT statements were never finalized before sqlite3_close_v2(), leaving it as a SQLite "zombie" until the whole object was garbage collected. Fixed, and verified on Windows/macOS/Linux CI with reproducer tests isolating each half of the bug.

v0.1.14

Choose a tag to compare

@jeandet jeandet released this 14 Sep 15:48

Fix #13: close() now releases the mmap LRU cache, same as clear() does. Previously a file-backed value read before close() stayed memory-mapped, which on Windows could make a post-close() shutil.rmtree() of the cache directory silently leave those files behind.

v0.1.13

Choose a tag to compare

@jeandet jeandet released this 11 Sep 18:18

Fixed

  • #12: with cache: silently closing the store on exit. The #11 fix (adding Cache.close()) also wired __exit__ to call close() — an unrequested addition, since diskcache's own close() is a cheap per-thread connection reset that reopens lazily on next access, unlike ours which is a real one-way shutdown. Reusing a Cache/Index/FanoutCache/FanoutIndex object across sequential with blocks silently returned None from get() instead of the stored value. with cache: is now a no-op again, matching pre-#11 behavior.
  • Any operation on an already-closed store (get, set, transact, etc.) now raises RuntimeError instead of silently misbehaving, so a real misuse is loud instead of returning wrong data.

No performance impact: benchmarked before/after on identical storage (get/set/transact micro-benchmark and the bgscan sustained-write benchmark) — results are within normal run-to-run noise.

v0.1.12

Choose a tag to compare

@jeandet jeandet released this 09 Sep 17:19

Full Changelog: v0.1.11...v0.1.12

v0.1.11

Choose a tag to compare

@jeandet jeandet released this 02 Sep 17:27

Full Changelog: v0.1.10...v0.1.11

v0.1.10

Choose a tag to compare

@jeandet jeandet released this 02 Sep 03:10

Full Changelog: v0.1.9...v0.1.10

v0.1.9

Choose a tag to compare

@jeandet jeandet released this 01 Sep 20:05

Full Changelog: v0.1.8...v0.1.9

v0.1.8

Choose a tag to compare

@jeandet jeandet released this 31 Aug 09:19

Eliminates the cross-process write race that caused spurious cache misses under concurrent access.

When one process overwrote a file-backed value (> 8 KB) while another was reading the same key, the writer could unlink the old blob file in the tiny window between the reader fetching its path and opening it — the reader then saw a miss (with an "Error loading file for key ... deleting entry" log line) for a key that was never deleted. The bounded retry added in 0.1.4 made this rare; deployments with hot same-key rewrite bursts still hit it regularly (~13×/hour observed on a production speasy-proxy).

Displaced files are now never unlinked inline. Their path is recorded in a new trash table in the same transaction as the row change, and the background thread removes them only after a 5-second grace period — far longer than any reader's microsecond exposure window. Existing databases migrate automatically on first open.

The same mechanism fixes three adjacent issues: incr() on a file-backed entry leaked the displaced file permanently; a crash between commit and unlink orphaned the old file; and a failed final flush in DiskStorage could silently leave a truncated file behind a committed row (writes now verify the flush).

Performance is unchanged or slightly better: the writer no longer performs a synchronous unlink on overwrite (the trash insert is cheaper), reads are untouched, and all operations remain faster than diskcache in the bundled comparison tests. Contract pinned by a new test suite, tests/python/test_trash_deferred_delete.py.