Releases: SciQLop/Sciqlop-cache
Release list
v0.2.1
First release since 0.1.16 on PyPI. It includes the 0.2.0 work (tagged, never published) and the fixes below.
Highlights
- Free-threaded Python (3.13t / 3.14t) is supported and tested. The extension runs without the GIL, and a new multi-threaded torture test runs in CI on 3.14t.
- Fixed a GIL deadlock on regular Python. A thread inside
transact()(or iterating) plus another thread callingiterkeys,incr,get_meta,check,clear,size,evict,stats, ... could freeze the interpreter. Every binding that takes the store lock now releases the GIL. - Fixed data loss on rollback.
delete,poporevict_tagof a large (file-backed) value inside atransact()that rolled back lost the value. Value files are now queued for deletion in the same transaction as the row change, so a rollback keeps them. - Fixed orphan files on Windows. A file still mapped by a live buffer could not be deleted and was left behind. It is now retried in the background until it can be removed.
- Fixed a crash: dropping a
FanoutCache/FanoutIndexwhile iterating over its keys segfaulted.
diskcache compatibility (from 0.2.0)
- Non-
strkeys,retry=on every method,get(tag=True, expire_time=True), diskcache constructor kwargs (directory=,size_limit=,shards=, ...). Indexis aMutableMapping;cache[missing]/Index.pop(missing)raiseKeyError;Timeoutexception.touch()matches diskcache: optionalexpire, no resurrection of expired keys,Falseon a miss,intexpire accepted (#14).
Performance
Measured against the previous commit, same machine, interleaved runs:
- Deleting large values is faster:
del-28%,pop-19%,evict_tag-84% per row. Files are now removed by the background thread, so their disk space comes back about 5 s later. set-5%, batchedset-11%,getunchanged.- The README benchmarks are refreshed, and the Performance section now documents the test setup.
Build
- nanobind 2.15, SQLite 3.53.4, Catch2 3.16, robin-map 1.4.1.
cpp_utilsis pinned to a fixed commit instead of trackingmain.
Full changes: v0.1.16...v0.2.1
v0.1.16
v0.1.15 fixed close() leaking mmap handles and the main sqlite3 connection on Windows (issue #13), but introduced a heap-use-after-free in Database::close() in the process: sqlite3_close_v2() now fully frees the connection once BEGIN/COMMIT are finalized, and close() then called db.reset(), whose deleter called sqlite3_close() again on that freed pointer. That's what crashed test_wheels(windows-latest, 3.14t) in CI ("Windows fatal exception: access violation").
v0.1.15 never reached PyPI -- its release workflow failed before the upload step, so nothing broken was published.
Fixed here (db.release() instead of db.reset()) and verified deterministically with AddressSanitizer: the bug reproduced on the very first close() call before the fix and is clean after it, across the full C++ suite and all 146 Python interface tests.
v0.1.15
Follow-up to v0.1.14 for issue #13. The previous fix only released the mmap LRU cache in close(); the main sciqlop-cache.db connection itself still leaked on Windows because Database's own BEGIN/COMMIT statements were never finalized before sqlite3_close_v2(), leaving it as a SQLite "zombie" until the whole object was garbage collected. Fixed, and verified on Windows/macOS/Linux CI with reproducer tests isolating each half of the bug.
v0.1.14
Fix #13: close() now releases the mmap LRU cache, same as clear() does. Previously a file-backed value read before close() stayed memory-mapped, which on Windows could make a post-close() shutil.rmtree() of the cache directory silently leave those files behind.
v0.1.13
Fixed
- #12:
with cache:silently closing the store on exit. The #11 fix (addingCache.close()) also wired__exit__to callclose()— an unrequested addition, since diskcache's ownclose()is a cheap per-thread connection reset that reopens lazily on next access, unlike ours which is a real one-way shutdown. Reusing aCache/Index/FanoutCache/FanoutIndexobject across sequentialwithblocks silently returnedNonefromget()instead of the stored value.with cache:is now a no-op again, matching pre-#11 behavior. - Any operation on an already-closed store (
get,set,transact, etc.) now raisesRuntimeErrorinstead of silently misbehaving, so a real misuse is loud instead of returning wrong data.
No performance impact: benchmarked before/after on identical storage (get/set/transact micro-benchmark and the bgscan sustained-write benchmark) — results are within normal run-to-run noise.
v0.1.12
v0.1.11
v0.1.10
v0.1.9
v0.1.8
Eliminates the cross-process write race that caused spurious cache misses under concurrent access.
When one process overwrote a file-backed value (> 8 KB) while another was reading the same key, the writer could unlink the old blob file in the tiny window between the reader fetching its path and opening it — the reader then saw a miss (with an "Error loading file for key ... deleting entry" log line) for a key that was never deleted. The bounded retry added in 0.1.4 made this rare; deployments with hot same-key rewrite bursts still hit it regularly (~13×/hour observed on a production speasy-proxy).
Displaced files are now never unlinked inline. Their path is recorded in a new trash table in the same transaction as the row change, and the background thread removes them only after a 5-second grace period — far longer than any reader's microsecond exposure window. Existing databases migrate automatically on first open.
The same mechanism fixes three adjacent issues: incr() on a file-backed entry leaked the displaced file permanently; a crash between commit and unlink orphaned the old file; and a failed final flush in DiskStorage could silently leave a truncated file behind a committed row (writes now verify the flush).
Performance is unchanged or slightly better: the writer no longer performs a synchronous unlink on overwrite (the trash insert is cheaper), reads are untouched, and all operations remain faster than diskcache in the bundled comparison tests. Contract pinned by a new test suite, tests/python/test_trash_deferred_delete.py.