Skip to content

Releases: rustic-ai/uni-db

Release v3.0.1

Choose a tag to compare

@milliondreams milliondreams released this 14 Jul 18:04
0812a49

What's Changed

Other Changes

  • Implement Plugin Compute ABI phases 0-4 and update version to 3.0.1 by @milliondreams in #148

Full Changelog: v3.0.0...v3.0.1

What's Changed

Other Changes

  • Implement Plugin Compute ABI phases 0-4 and update version to 3.0.1 by @milliondreams in #148

Full Changelog: v3.0.0...v3.0.1

Release v3.0.0

Choose a tag to compare

@milliondreams milliondreams released this 13 Jul 04:38
9b151bb

Release focus: a plugin-framework honesty pass — the advertised plugin API is subtracted down
to what the engine actually honors (four dead traits removed, one ABI changed) — landed alongside
the release's headline capability, GraphCompute: guest-authorable graph algorithms in Rhai,
Python, WASM, and Extism. Rounding out the release: real firing triggers, configurable
full-text tokenizers/analyzers
, a BINARY_VECTOR type with exact Hamming/Jaccard, new
hybrid-search fusion methods, relationship and composite-key UNIQUE constraints, Locy
generator predicates
, and a large multi-wave correctness-scan hardening pass.

This is a major release covering everything since 2.5.0: 139 commits, version bumped
across the Rust workspace and the Python packages (uni-db, uni-pydantic) to 3.0.0. It is
numbered 3.0.0 — not 3.5.0, despite the internal development history bumping through 3.1–3.5 while
features landed — because it carries breaking removals of plugin surfaces; all breaking changes
are consolidated under this single major tag. If your code does not implement custom uni-plugin
traits, it should recompile unchanged. See Migration guide and Upgrade notes below.


⚠️ Breaking changes

This release makes the plugin API match what the engine dispatches. Four registrable-but-never-invoked
traits are removed, and the trigger ABI changes to enable real firing.

Removed plugin surfaces

  • PregelProgramProvider (traits::algorithm) — a stub with no executor; never invoked. Its
    support types (PregelSignature, AggregationMode, ComputeOutcome, PregelStats) and the
    .pregel(..) registrar are removed too. Author against AlgorithmProvider (retained), or use
    the new GraphCompute engine for guest-authored graph algorithms.
  • OperatorProvider (traits::operator) — custom physical operators were never inserted into
    the planner. Removed with its .operator(..) registrar. Use the retained OptimizerRuleProvider
    (rule() / physical_rule()) for planner extension.
  • Plugin StorageBackend (traits::storage) — the scheme-keyed durable backend was never
    consulted by the storage engine. Removed with .storage_backend(..). The per-label Storage
    surface (label_storage() / lookup_label_storage) is retained. (The internal
    uni_store::backend::StorageBackend is a different, unaffected trait.)
  • Connector (traits::connector) — a lifecycle-only stub with no query-time data path.
    Removed, along with Capability::Connector, SurfaceKind::Connector, and the public
    Uni::start_connector / Uni::stop_connector / ConnectorLifecycle API. For external
    data, use CatalogProvider / ReplacementScanProvider. AuthProvider / AuthzPolicy
    are retained.

The shared capabilities Capability::{Operator, Storage, Algorithm} are retained — they back the
delivered optimizer_rule / label_storage / algorithm surfaces.

Changed ABI

  • TriggerContext now carries an owned private Option<Arc<dyn ProcedureHost>> with with_host()
    / host() accessors — this is what lets a declared trigger execute its Cypher action body.
    TriggerContext::new is unchanged (host defaults to None), so existing TriggerPlugin
    implementors keep compiling, but the struct's shape/size changed: recompile custom plugins
    against 3.0.0
    ; do not mix ABIs across the boundary.
  • FireMode::EventualConsistency now batches for real instead of aliasing Async. Events
    coalesce per-trigger and drain on an interval/size threshold (UniConfig::ec_flush_interval,
    default 1 s; ec_flush_threshold, default 10,000). If you relied on immediate per-event firing,
    set Async explicitly or tune the thresholds.

Highlights

🔹 GraphCompute — guest-authorable graph algorithms (Rhai · Python · WASM · Extism)

Third parties can now author graph algorithms — PageRank / Personalized PageRank, reachability/BFS,
WCC, Bellman–Ford, k-core, eigenvector, HITS/Katz, random walks, neighbourhood similarity — as
plugins, with no forking and no shipping Rust.

The model is "conductor, not worker": the guest runs only the O(iterations) control loop while
native code does all O(V+E) work, so only opaque handles and scalars cross the plugin boundary
never frontiers, neighbour lists, or property columns. Because nothing heavy is marshalled, the same
design runs uniformly across all four loaders (Rhai, Python, WASM Component Model, Extism); each
ships a Personalized PageRank example that matches the native provider to 1e-9.

  • Kernel catalog — a fixed set of coarse native kernels: frontier/expand (direction-optimized,
    mask-fused), SpMV over named semirings, map/reduce/scatter, set ops, arg-extreme / top-k, random
    walks, all-pairs / neighbourhood overlap (Jaccard, cosine, Adamic–Adar, triangle count), and
    result emit (including ragged walk and pairwise egress). Values are Arrow-backed; handles are
    generational, epoch-tagged, kind-checked, and session-scoped.
  • Deterministic & fail-closed — deterministic CSR ordering and fixed-order reductions give
    bitwise-reproducible results across thread counts. A native-work budget and a per-session
    handle-memory arena are fail-closed, and non-convergence is a hard error (GraphComputeIncomplete
    with distinct Exhausted / IterationLimit / Timeout reasons). Guest loops are bounded per loader
    (Rhai catch_unwind, a Python KeyboardInterrupt watchdog, WASM/Extism epoch interruption).
  • Typed args & version negotiation — signatures declare typed, defaulted arguments (arity- and
    type-checked, default-filled at call time) and negotiate host kernel-slice versions at registration.
  • First-party providers uni.algo.gcpagerank, uni.algo.gcwalks, and uni.algo.gcoverlap dogfood
    the surface. Design: docs/proposals/graphcompute_plugin_api_2026-07-10.md.

🔹 GraphView topology API

A slot-indexed, read-only topology trait (mirroring the internal CSR GraphProjection) plus
AlgorithmHost::project(), gated on Capability::HostQuery. This is the in-process path for
authoring graph algorithms against public traits, and AlgorithmProvider::run is now wired into
CALL dispatch on both the planner and simple-executor paths (miss-only, so built-in uni.algo.*
never regress). First-party uni.algo.reachability, uni.algo.pagerank, uni.algo.sssp, and
uni.path.expand (an APOC-style bounded path expander) are authored purely against this surface.

🔹 Real triggers — declareTrigger that actually fires

uni.plugin.declareTrigger now installs a real firing TriggerPlugin (AfterCommit / Async) instead
of a callable procedure that never fired. Declared triggers support an event filter
(CREATE|UPDATE|DELETE [ON :Label | -[:Type]-] [WHEN pred] [ASYNC]), bind event columns
($vid / $label / $event_kind), run the declared Cypher action body, and replay across
restart
. FireMode::EventualConsistency now provides a real batched queue that coalesces
per-trigger events and drains on interval/size thresholds with lossless back-pressure.

🔹 Full-text tokenizer / analyzer configuration

Full-text indexes now honor tokenizer/analyzer/stemmer/stop-word configuration end-to-end:

CREATE FULLTEXT INDEX ... OPTIONS {
  analyzer, language, stemmer, stopwords,
  ascii_folding, lower_case, max_token_length, ngram_min, ngram_max
}

Previously every FTS index used Lance's default "simple" tokenizer regardless of configuration,
making CJK / multilingual text effectively unindexable. 18-language stemming and stop-words plus
ngram tokenization are now supported (CJK requires dictionary files under LANCE_LANGUAGE_MODEL_HOME).

🔹 BINARY_VECTOR(n) type with exact Hamming & Jaccard

Bit-packed binary embeddings stored at full fidelity (n = byte count, so BINARY_VECTOR(4) = 32
bits) — ideal for hash/fingerprint embeddings and set-overlap similarity.

CREATE LABEL Doc (bits BINARY_VECTOR(4));
CREATE (:Doc {bits: [0, 255, 165, 60]});
RETURN VECTOR_DISTANCE(a.bits, b.bits, 'hamming');   // or 'jaccard'

Brute-force exact only (binary metrics have no ANN index; attempting to build one is rejected with
guidance). Additive, no breaking change. Relatedly, VECTOR_DISTANCE is now a real DataFusion
scalar UDF
, so it works in DataFusion-planned RETURN clauses for all metrics, not just the
interpreter path.

🔹 Hybrid search: two new fusion methods

uni.search gains DBSF (distribution-based, z-score normalized) and relative-score (min-max

  • weighted) fusion, joining the existing rrf and weighted methods for fusing dense + FTS/BM25 +
    learned-sparse arms. Select via options.method; tune with rrf_k (default 60), weights, alpha,
    over_fetch. Also added: DBSF / relative-score exposure through the OGM, and L1/Manhattan and
    geo Point compute paths.

🔹 Relationship & composite-key UNIQUE constraints

  • Relationship (edge-type) UNIQUE / NODE KEY constraints with full flush-safe, false-positive-free
    write-horizon enforcement and a commit-time SSI guard.
  • Composite node-key constraints ((a, b) IS UNIQUE / IS NODE KEY) enforced end-to-end.

🔹 Locy: fixed-arity generator predicates

Plugin-registered table-valued predicates that bind new variables and explode one source row into
many — e.g. myplugin.range(n.k) -> (i) yields i ∈ {0,1,2} for k = 3. The -> arrow
distinguishes a generator from a filter predicate; bindings flow through YIELD like ALONG, and
multiple generators compose per body. (Deferred: variable arity, recursion-through-generator, and
feeding FOLD.) This work also fixed a latent bug where the plugin registry was never task-scoped on...

Read more

Release v2.5.0

Choose a tag to compare

@milliondreams milliondreams released this 03 Jul 19:14
155e4e7

Release focus: typed 3-way hybrid search in the Python OGM, a query-time HNSW ef_search
knob backed by at-scale recall benchmarks for all three vector modalities, capability-based
embedding-alias routing (one BGE-M3 alias can now serve dense, sparse, and multi-vector columns),
and a query-planner correctness/performance pass (equi-join recovery, projection-pushdown leaks).

This is a minor release covering everything since 2.4.1. 29 commits; version bumped across the
Rust workspace and the Python packages (uni-db, uni-pydantic) to 2.5.0. New public surfaces:
the OGM hybrid_search() builder and .search_scores sidecar, the ef_search option on
uni.vector.query, and UniConfig::flush_stream_timeout. API-boundary behavior changes are listed
under Upgrade notes.


Highlights

🔹 OGM hybrid_search() builder + .search_scores sidecar — #114

The engine's three-way fused retrieval (uni.search: dense + full-text + sparse) is now reachable
from the typed Pydantic OGM instead of hand-written Cypher.

  • Model.query().hybrid_search(vector=, fts=, sparse=, method=, weights=/alpha=, k=, ...) emits
    the 7-positional uni.search call. All user data is bound as $params; only schema identifiers
    and numeric knobs are interpolated, and every resolved property name passes a format-only
    validation before it reaches the query string (closing an injection surface across all three
    search builders).
  • .search_scores sidecar. Results carry fused + per-arm scores via a UniNode private
    attribute, so scores survive hydration under extra="forbid" and never collide with a user field
    named score (the elasticsearch-dsl .meta.score pattern, done Pydantic-correctly).
    vector_search and sparse_search were retrofitted with the same sidecar for a uniform scored
    surface.
  • Latent hydration bug fixed. vector_search/sparse_search previously emitted
    RETURN node AS n, which cannot hydrate through the OGM's row decoder — their .all() silently
    returned []. Both now return the properties()/id()/labels() triple plus score columns, with
    execution tests (including previously-absent async coverage) locking the surface in.
  • Design and ecosystem survey: docs/proposals/ogm_hybrid_search.md. Package suite grew 240 → 257
    tests.

🔹 Query-time HNSW ef_search + at-scale recall benchmarks

New CI-runnable criterion recall benches give all three modalities (sparse / dense / multi-vector)
an oracle-vs-engine recall+latency benchmark through the public uni.vector.query path
(benches/dense_retrieval.rs, benches/multivec_retrieval.rs; sparse already existed).

  • The dense bench surfaced a real product gap: the HNSW search beam width was never plumbed to
    query time, so it sat at lancedb's 1.5·k default and recall@10 collapsed to 0.300 at 10k
    docs
    . ef_search (alias ef) is now accepted in the uni.vector.query options map and
    threaded through VectorQueryOpts.ef to the index search.
  • With a wide beam, dense HNSW recall@10 recovers 0.300 → 1.000 at 10k docs at essentially
    unchanged latency (6.1 → 6.6 ms).

🔹 Capability-based embedding alias routing — #129, #130

Routing and open-time validation assumed one task per alias, so a hybrid model (BGE-M3,
EmbedHybrid) could not serve a plain vector column on its own alias.

  • Validation (#130). The open path rejected any non-Embed task on a Vector index — a dense or
    multi-vector index on an EmbedHybrid/EmbedMultiVector alias created fine but failed to
    reopen, and sparse aliases were never validated at all. The blanket check is replaced with a
    capability test (required_heads ⊆ text_embedding_heads(task)) across Vector and Sparse indexes,
    naming the offending column on failure. Image/audio/multimodal embed tasks map to no text heads,
    so binding a text auto-embed column to them is rejected at open.
  • Routing (#129). A lone head routed to the narrow single-task facade, which hybrid models do
    not implement. Lone-head requests now fall back to the hybrid embedder for the requested head —
    with no double model load, and a head the model does not expose remains a hard error, never a
    silently-empty column.
  • The shared capability model lives in a new uni-store module (embed_caps), with a unit matrix
    plus end-to-end alias-capability tests. Design: docs/proposals/embedding_alias_capability_model.md.

🔹 Equi-join recovery: HashJoin instead of Cartesian products — #131

A recursive Locy IS-ref join degraded to a quadratic CrossJoinExec because the join-recovery
variable collector silently dropped LocyDerivedScan behind a _ => {} wildcard — the derived
relation's columns were invisible, so equality conjuncts were never recognized as equi-join pairs.
The same defect was reachable from Cypher via named-path cross-MATCH equi-joins (BindPath).

  • The collector (and five sibling plan walkers with the same latent bug class) are now exhaustive:
    a new LogicalPlan variant fails to compile until it is classified.
  • Node scans register their _vid key form so IS-ref equality keys match exactly; the hash-join
    recovery path mirrors the CrossJoin lowering's structural-column stripping.
  • Regression tests prove CrossJoinExec rows now scale linearly, and HashJoin recovery is
    exercised for BindPath, VectorKnn, ShortestPath, AllShortestPaths, and
    BindZeroLengthPath — each shown to go red on a blinded collector.

🔹 Projection-pushdown "*" leaks closed — #134

A dense similar_to scan ran ~60× slower when scanned rows carried an unread List(Vector)
column: a bare entity reference marked the scan as needing all properties ("*"), decoding wide,
unused columns per row. The whole leak family is closed:

  • Scalar functions over entities (id/elementId/type/count/startNode/endNode) no longer
    re-add "*" through a second recursion into their arguments.
  • count(DISTINCT n) dedups on the entity identity column instead of the full struct;
    collect(DISTINCT n) dedups by identity for entity maps — fixing a latent HashMap-order
    nondeterminism that meant it previously never actually deduped.
  • WITH n / WITH n AS m narrow the forwarded entity to the properties actually accessed
    downstream; returned-whole entities, rename chains, and non-narrowable kinds (paths) stay wide to
    avoid silent NULLs.

🔹 Traversal reads survive the L0 → Lance flush — #135 and the schemaless _all_props fix

Two related property-materialization gaps, both reported as mysterious NULL/empty reads:

  • #135. The single-hop traverse operator materialized target-vertex properties from the L0
    in-memory buffers only. Once the ~5s auto-flush migrated rows to Lance, every target property of
    MATCH (a)-[:REL]->(b) RETURN b.prop read back NULL — deterministic, not the "race under CPU
    load" the report hypothesized. Targets are now prefetched through the property manager, which
    merges L0 + persisted storage under the query's MVCC visibility.
  • Schemaless _all_props. In schemaless mode the "all properties" wildcard collapsed to an
    empty name list (it enumerated schema-declared names — none exist without a schema), so multi-hop
    traversal targets and whole-node projections returned empty {} nodes. The wildcard sentinel is
    now threaded through the batch property helpers, which also gained a main-table (props_json)
    fallback for flushed schemaless vertices. Fixes the schemaless-lane openCypher TCK
    MatchWhere6 [7]/[8] and Locy TCK AssumeAbduce failures.

🔹 Traversal label resolution survives the L0 → Lance flush — the _labels family, #141, #140

A vertex's labels, like its properties (#135), leave the in-memory L0 buffers once it is flushed
to Lance and live only in the persisted VidLabelsIndex. Three label-resolution seams read L0 only
and misbehaved after a flush (the ~5 s auto-flush, an explicit db.flush(), or a fork, which
flushes all data before branching):

  • Output columns (the _labels family). Three traversal output builders — the single-hop sync
    fast-path and both variable-length-path builders (schema-aware and schemaless) — resolved a
    target's _labels column from L0 only, so labels(m) / hasLabel(m, …) over a flushed or forked
    vertex returned an empty label set. All three now route through a shared per-row resolver
    (resolve_vertex_labels: L0 chain then persisted index), making every current and future
    traversal label-column correct by construction; a nullable-aware builder covers unmatched VLP rows.
  • VLP label filters over-admitted (#141). The variable-length-path predicates
    check_target_label (terminal (m:Label)) and check_state_constraint (QPP intermediate
    (y:Label)) read L0-only labels and failed open — admitting a flushed vertex without checking
    its label. MATCH (a)-[:R*1..n]->(m:Label) and QPP intermediate-label constraints could return
    paths through vertices that don't carry the required label. Both predicates now evaluate against
    the vertex's real labels via resolve_vertex_labels. Behavior-changing: affected VLP/QPP
    queries now reject rows they previously over-admitted.
  • Deleted-vertex label resurrection (#140). resolve_vertex_labels consulted the persisted
    index without checking tombstones, so a flushed-then-deleted vertex — its tombstone recorded in a
    live L0, its index entry not yet re-flushed — could resolve to its stale labels. The resolver
    now returns an empty label set for a tombstoned vid, so every label predicate rejects the deleted
    vertex instead of admitting it through a fail-open path. (End-to-end this is masked by cascade
    edge-tombstoning; the guard closes the seam at the resolution chokepoint.)

Each fix ships with tests verified red-before / green-after...

Read more

Release v2.4.1

Choose a tag to compare

@milliondreams milliondreams released this 28 Jun 02:23
35936bb

Release focus: learned-sparse (SPLADE) vector search, a vector-index test-parity and
hardening pass across all three modalities (dense / multi-vector / sparse), and a storage-path
correctness fix.

This is a minor release covering everything since 2.3.0. 31 commits; version bumped across the
Rust workspace and the Python packages (uni-db, uni-pydantic) to 2.4.1. The feature work is
driven by a new public API (StorageBackend::lock_table_for_write + TableWriteGuard) and
API-boundary behavior changes (malformed sparse values now return a clean error / are canonicalized
instead of panicking; uni.sparse.query rejects an unsupported filter argument).

Note on 2.4.0. The v2.4.0 tag was cut but its release run failed immediately at the
version-validation gate — bindings/uni-pydantic/pyproject.toml had not been bumped in lock-step
with the Rust workspace, so no artifacts were ever published (no GitHub assets, no PyPI wheels).
2.4.1 is the first published release of this line and ships the identical feature set. The
version-sync process that prevents this is now documented in
docs/releasing_version_bump.md.


Highlights

🔹 Learned-sparse (SPLADE) vector search — #95

uni-db now has a first-class sparse / learned-sparse vector type and a scored inverted-index
retrieval path, completing the third vector modality alongside dense and multi-vector (ColBERT).

  • Type & crate. A new uni-sparse-vector leaf crate (BTIC-style) provides the SparseVector
    type with a validating constructor, a lossless binary codec, and pure sparse_dot / l2_norm /
    prune_top_k kernels (unit + property tested). Value::SparseVector / DataType::SparseVector
    lower to an Arrow Struct{indices: List<UInt32>, values: List<Float32>} with TAG_SPARSE_VECTOR
    CV-tag framing.
  • Scored index. A SparseVectorIndex (fork of the inverted index) stores per-term postings with
    a max_impact upper bound and serves a query_topk dot-accumulator + min-heap. Built via
    segment-merge (backend-scan backfill + L0-incremental flush), MVCC/tombstone-correct: sparse_rerank
    unions L0, gates on version / _deleted, and exact-dot rescores.
  • Surfaces. uni.sparse.query('Label', 'prop', query, k) procedure, SPARSE_VECTOR(N) Cypher
    type DDL, and CREATE VECTOR INDEX ... OPTIONS{type:'sparse'} index DDL (now correctly routed to
    the sparse path and backfilled after a flush — it previously built a dense IVF_PQ index silently).
  • 8-bit weight quantization (M3). Per-term unsigned UInt8 codes + a Float32 weight scale;
    max_impact is computed from the dequantized weights so it stays a valid rank-safety upper
    bound. A single reader detects the encoding by element type, so legacy f32 segments and
    quantize=false share one lossless path — no version marker, no rebuild.
  • Scoring / fusion / hybrid (M4). N-ary RRF (fuse_rrf_multi) and source-aware weighted fusion
    (fuse_weighted_sources with DistanceToSim / ScoreByMax normalization); a scalar
    sparse_similar_to(a, b); and a 3-way hybrid search (dense + full-text + sparse) emitting a
    sparse_score column. Two-way fusion is byte-identical (an empty source is a no-op).
  • Fork-aware. uni.sparse.query now works inside a fork (Approach A: brute-force branch scan
    over the inherited + fork rows, with the already-fork-correct re-rank unioning fork L0).
    Previously sparse search returned nothing on a fork.
  • Text auto-embed. Sparse columns auto-embed at write time, query time, and in hybrid search
    (xervo 0.17 sparse head); columns sharing an alias + source with a dense/multi-vector column are
    filled in a single hybrid forward pass. A cross-modality fix ensures a SET on an embed source
    column re-embeds the target (all modalities previously left a stale vector on SET).
  • Python / OGM. PySparseVector, DataType.sparse_vector(N), the SparseVector[N] OGM type
    with a sparse_search() builder and schema auto-indexing, and a fix for returned sparse
    properties being silently dropped (value_to_py had no SparseVector arm).
  • Performance. End-to-end uni.sparse.query (k=10, median): ~10 ms at 10k docs (int8 ~ f32),
    recall@10 = 1.000. Sub-15 ms at 10k confirms the P1 index needs no P2 block-max pruning at this
    scale, which is documented as deferred-by-design.

🔹 Vector-index hardening & test parity — #95/#96 adversarial review, #121, #122

An adversarial multi-agent review of the sparse (#95) and multi-vector/MUVERA (#96) features
surfaced 32 verified findings; the critical and high-severity ones are fixed here, each with a
regression test (report: docs/sparse_multivector_review_2026-06-27.md). In parallel, dense and
multi-vector were brought up to the sparse test suite's "gold standard", which uncovered two real
correctness bugs.

  • RC1 (critical data loss) — an unserialized Lance Overwrite vs flush race. Fixed with a new
    StorageBackend::lock_table_for_writeTableWriteGuard (owned per-table mutex guard); MUVERA
    FDE backfill and sparse/inverted posting backfill now hold the lock across scan → splice →
    atomic-replace, closing the read-then-overwrite TOCTOU window where a concurrent flush append was
    silently dropped.
  • RC2–RC6 — malformed SparseVector values are canonicalized / rejected with a clean error
    instead of panicking (ingest, WAL, and Arrow paths); FDE param integer-overflow guards
    (checked_mul/checked_shl + bounds); a wrong-dim multi-vector token now skips the row instead of
    wedging every flush; stale postings are purged on update-reflush; and table_exists() checks
    fail-closed (?) instead of fail-open across the sparse/multivector/muvera search and backfill
    paths.
  • Dense L0-union bug (real). uni.vector.query silently ignored committed-but-unflushed L0 data
    — dense search returned stale results until a flush. Root cause: the L0 candidate scorer used
    Value::as_array(), which matches only Value::List and dropped the typed Value::Vector the
    Cypher write path actually stores. Now extracted via the canonical TryFrom<&Value> for Vec<f32>.
  • RETURN-projection type fidelity. A top-level FixedSizeList<Float32> now decodes to
    Value::Vector (not a generic Value::List), restoring type identity on RETURN d.vec_col for
    dense and multi-vector (parity with SparseVector); VECTOR_SIMILARITY/VECTOR_DISTANCE/=~ were
    routed through a helper accepting both variants. Also fixes an L0 multi-vector projection
    hard-error.
  • Prebuilt-runtime reopen. Opening a DB whose schema has embedding-configured indexes no longer
    errors "catalog is required" when a prebuilt runtime (.xervo_runtime(...)) — which carries its
    own catalog — is attached.
  • Parity suites. Dense and multi-vector now match sparse on every axis: brute-force exact oracles
    (cosine-KNN, MaxSim), crash/WAL failpoint resilience, MVCC snapshot isolation, and metamorphic
    oracles (PR smoke + nightly soak). The #121 partial-Lance-write report was investigated and found
    to be a stale test assertion, not a production bug (regressions added). Residual auto-embed cells
    are tracked in #122.

🔹 Storage-path correctness — #115, #116, #117

VertexDataset::new built {base}/vertices_<label> while the LanceDB backend stores
{base}/vertices_<label>.lance, so raw-path reads of flushed vertex tables silently returned
Err/0 rows. Two confirmed consequences: composite-key uniqueness failed open across the flush
boundary (#116), and scalar/inverted/FTS/vector index builds were silently skipped after a flush
(#115) — masked by full-scan / brute-force fallbacks, which is why result-asserting tests never
caught it (#117).

  • The canonical .lance path is now built from a single source of truth.
  • All index creation routes through StorageBackend and the VertexDataset raw-open escape hatch
    (open/open_at/open_raw/new_branched) is deleted — there is now one on-disk path
    reconstruction, so the two paths can no longer drift. This eliminates the bug class and unifies
    the primary index path with the fork path. The trait gained
    create_vector_index/create_scalar_index/create_fts_index/drop_index with full param
    fidelity against lancedb 0.30.
  • Mechanism tests (via list_indexes) now assert the index was physically built, not just that
    query results are correct — proven to catch the bug on revert.

🔹 Durability — crash-during-flush

A panic at the flush seam followed by a graceful close (Drop → shutdown flush) silently dropped an
acknowledged, committed-but-unflushed transaction. The next flush truncated the pending buffer's own
WAL segment and advanced the checkpoint past it — both keyed off the buffer's high watermark
instead of its start watermark. Fixed by tracking wal_lsn_at_start per L0 buffer and capping both
WAL truncation and the published high-water-mark at the floor over all other pending buffers.
(Under a real, Drop-less crash the WAL stayed durable, so only the graceful-close path lost data.)

🔹 Locy (Datalog) typed-value boundary — #111, #112, #113

Three distinct root causes, unified by typed values losing their logical type when crossing the
Locy ↔ DataFusion boundary:

  • #112 — a property-expression KEY (i.tag AS tag) projected NULL without a FOLD; the
    QUERY/SLG path now evaluates KEY expressions whenever one needs evaluation.
  • #111 — duration arithmetic (duration.inDays(...) + math) crashed at plan time
    ("Unsupported CAST from LargeBinary to Float64") and poisoned all rules; math now routes through
    cypher_to_float64_expr so non-numeric values yield a clean NULL. Numeric extraction via
    .days/.months/.seconds.
  • #113 — th...
Read more

Release v2.0.0

Choose a tag to compare

@milliondreams milliondreams released this 04 Jun 06:38
53820c4

Uni 2.0.0 Release Notes

Release scope: v1.1.02.0.0 · 283 commits · ~245K insertions across 1,261 files
Dates: 2026-04-15 → 2026-06-03
Version path: 1.1.01.1.11.2.01.3.0 (breaking wheel collapse) → 1.4.02.0.0

Uni 2.0 is the largest release in the project's history. It graduates four headline subsystems from prototype to default-on production behavior — Serializable Snapshot Isolation, Graph Forks, the Plugin Framework, and Locy neural predicates — alongside a sweeping query-planner performance overhaul, an asynchronous write path, and a refreshed dependency baseline (LanceDB 0.30 / Lance 7 / Arrow 58 / DataFusion 53, plus every workspace dependency brought to latest).


⚠️ Breaking & Behavioral Changes (read first)

Two changes can affect existing deployments. Neither is an API-signature break — which is exactly why they need attention.

1. Concurrency model: Last-Writer-Wins → Serializable Snapshot Isolation (default-on)

This is a silent behavioral change: your code compiles and runs unchanged, but concurrent writers now behave differently.

  • Before (1.x): Two transactions writing the same vertex/edge concurrently → one write silently overwrote the other (lost update). Concurrent MERGE on the same unique key could create duplicates.
  • Now (2.0): Conflicting concurrent commits abort with a retriable error (SerializationConflict / ConstraintConflict) instead of silently losing data.

Migration:

  • For correctness, no action is required — you are now safe from lost updates by default.
  • Workloads with concurrent writers should wrap transactions in retry logic (see Concurrency below for transact_with_retry / execute_with_retry).
  • To restore exact 1.x semantics, set UniConfig.ssi_enabled = false (opt-in, not recommended; logs a warning when FOR UPDATE is used).
  • The compile-time ssi and l0-snapshot cargo features have been removed — SSI is always compiled and toggled at runtime.

2. Python wheel matrix collapsed: 12 → 6 wheels (v1.3.0)

Three GPU/provider axes were consolidated. Several wheel package names no longer exist.

Removed Replacement
uni-db-fastembed* (×3) uni-db (ONNX provider; FastEmbed alias strings unchanged)
uni-db-mistralrs* (×3) folded into uni-db / uni-db-cuda
uni-db-all* (×3) uni-db (CPU, all 11 providers) / uni-db-cuda (GPU)

Surviving wheels: uni-db, uni-db-onnx, uni-db-cuda, uni-db-metal, uni-db-onnx-cuda, uni-db-onnx-metal.

Provider migrations:

  • local/fastembedlocal/onnx; local/onnx-rerankerlocal/onnx (reranking unified under the ONNX provider).
  • Embed cache path moved: .uni_cache/fastembed/<alias>/.uni_cache/onnx-embed/<sanitized-repo>/.
  • Seven retired GPU execution providers (gpu-tensorrt, gpu-rocm, gpu-coreml, gpu-directml, gpu-openvino, gpu-qnn, gpu-wgpu) remain reachable via --features provider-onnx-dynamic + ORT_DYLIB_PATH.

Full migration guide: docs/migrations/0.9.0-wheel-matrix-collapse.md.


🌟 Headline Features

Serializable Snapshot Isolation & OCC (default-on)

Uni now provides serializable transaction isolation via Snapshot Isolation plus Optimistic Concurrency Control, replacing Last-Writer-Wins as the default.

  • Conflict detection is both write-write (two transactions touching the same item) and read-write/SSI (a committed write touches an item the aborting transaction read). Read-set tracking records vertices, edges, and neighbor traversals — including post-filter scan reads.
  • Retry helpers:
    • UniError::is_retriable() classifies transient contention vs. permanent failure.
    • Session::transact_with_retry(closure, RetryOptions) and execute_with_retry(query, …) automatically re-run a fresh transaction on retriable errors with jittered exponential backoff (default 5 attempts, 200µs base, 50ms cap).
  • FOR UPDATE (pessimistic escape hatch): Acquiring FOR UPDATE on a fresh transaction re-pins the snapshot to the latest committed state, enabling zero-retry read-modify-write:
    MATCH (n:Counter {id: 1}) FOR UPDATE
    SET n.count = n.count + 1
  • CRDT carve-out: Concurrent increments to CRDT-declared properties (COUNT, SUM, …) merge at commit time instead of aborting — provided the committed value is the same CRDT variant (a variant mismatch aborts to prevent silent loss).
  • Config: UniConfig.ssi_enabled (default true).

Known limitations: phantom reads from bare label scans are not tracked (guard scan-based RMW with FOR UPDATE); FOR UPDATE re-pinning only applies to fresh transactions; read-only session.query() paths observe their begin snapshot.

Graph Forks

Named, durable, isolated, writable graph branches — derived from a primary database or a parent fork via Lance copy-on-write, for safe experimentation, staging, audit, and what-if workflows without touching production data.

  • Lifecycle: session.fork(name) (open-or-create; .new_() to require fresh; .ttl(duration) for wall-clock expiration). Admin via list_forks(), fork_info(), drop_fork(), drop_fork_cascade().
  • Nested forks: children inherit parent state and read through the parent→child branch chain.
  • Governance: TTL with a background sweeper, a max_forks budget (ForkBudgetExceeded), tagging for GC-exempt regulatory holds (tag_fork / untag_fork / list_fork_tags), and parent→child cancellation cascades.
  • Fork-local schema: forked_session.fork_schema().label(…).apply() grows schema in the fork only, persisted in an overlay file; the primary is unaffected.
  • Structural diff & promotion:
    • diff_fork_primary(name) / diff_forks(a, b)ForkDiff of added/deleted/changed vertices and edges, paired by content-addressed UID (SHA3-256 of label+properties) so identity is correct even across VID collisions.
    • promote_from_fork(name, &[PromotePattern]) bulk-promotes matched rows back to primary in one atomic transaction, with a PromoteReport of inserted/skipped counts. Patterns support PromotePattern::label(...) / ::edge_type(...) with an optional .where_clause(...).
  • Fork-local indexes: vector (IVF-Flat) and full-text (BM25 RRF) indexes built over fork branches, with lossless (BTree union, k-way merge) and lossy (ANN/BM25 RRF) fusion against the parent — visible in explain() as FusedIndexScan.
  • Python: full sync + async bindings for the entire fork surface.

Known limitations: the shared UID→VID index dataset is not branch-isolated (lookups verify against the primary session); adding a new property column to an existing primary label is unsupported on active forks (drop-and-recreate the fork); parallel edges of the same type between the same endpoints currently share a UID.

Plugin Framework

A single capability-gated PluginRegistry replaces five ad-hoc registries. Every built-in (vector indexes, Lance storage, Locy aggregates, APOC procedures) is itself a plugin.

  • Five loaders: native Rust (all 23 extension surfaces, zero-cost), WASM Component Model (wasmtime), Extism, Rhai (sandboxed scripting), and PyO3 (in-process Python, vectorized eval). Sandboxed loaders currently expose scalar / aggregate / procedure surfaces.
  • Capability & security model: a CapabilitySet gates both extension surfaces and host imports (network, filesystem, host-query, KMS, secrets, locks, config, plugin-storage), each with attenuation (glob patterns, scopes, key/secret allowlists) and resource quotas (memory, fuel, wall-clock, concurrency, result rows). Effective capabilities = declared ∩ granted, enforced at three layers: registrar gate, WASM linker (ungranted host fns are structurally omitted), and runtime argument checks.
  • Trust & manifests: PluginManifest carries id/version/ABI/capabilities/determinism/side-effects plus a Blake3 hash pin (always verified) and an optional Ed25519 signature verified against a TrustRoot. Host trust policy (Disabled / WarnIfUnsigned / RequireSigned) is set via .plugin_trust(...).
  • APIs: Rust add_plugin / load_wasm_component / load_wasm_extism / load_rhai_plugin / load_python_plugin; Python db.load_wasm_component(...) / load_wasm_extism(...) / load_rhai_plugin(...) / load_python_plugin(...) plus @db.scalar_fn / @db.aggregate_fn / @db.procedure decorators, all with sync + async parity. Each load returns an outcome dict (effective/denied capabilities, registered surfaces).
  • Worked example: geo.haversine ships for all five loaders in examples/, with cross-loader byte/ULP parity tests.

Security fixes during the cycle: closed a WASM secret-handle bypass in decode_batch and a Rhai filesystem path-traversal escape.

Locy Neural Predicates & Probabilistic Reasoning

Locy gains DeepProbLog-class neuro-probabilistic reasoning operating natively on the property graph.

  • Neural classifiers: declare learned classifiers with CREATE MODEL … INPUT … FEATURES … OUTPUT PROB|SCORE|LABEL|VECTOR … USING xervo('provider') CALIBRATION … VERSION …, then invoke them in rule bodies (WHERE fraud_detector(inv) > 0.7, YIELD … AS risk PROB). Python callables can be registered directly as classifiers via config.register_classifier(alias, fn) / classifier_registry={…}.
  • FEATURE expressions: node properties, embeddings, hybrid retrieval (semantic_match(prop, 'query'), similar_to(...) with auto-embed), graph topology (degree_centrality, pagerank_score, betweenness_centrality, and more), neighbor aggregation (avg_neighbor/max_neighbor/sum_neighbor with direction), and path-context values carried from prior rule derivations.
  • **Calibra...
Read more

Release v1.1.0

Choose a tag to compare

@milliondreams milliondreams released this 15 Apr 20:56
6efbb2f

What's Changed

Other Changes

  • Integrate BTIC support in query engine, Python bindings, and tests by @milliondreams in #38

Full Changelog: v1.0.2...v1.1.0

What's Changed

Other Changes

  • Integrate BTIC support in query engine, Python bindings, and tests by @milliondreams in #38

Full Changelog: v1.0.2...v1.1.0

Release v1.0.1

Choose a tag to compare

@github-actions github-actions released this 10 Apr 07:41
9606bfe

What's Changed

Other Changes

  • fix: restore notebook execution outputs in documentation build by @milliondreams in #21
  • Fix FOLD projection issues, add serialization, and improve timeout handling by @milliondreams in #26
  • Fix FOLD projection issues, add serialization, and improve timeout handling by @milliondreams in #28

Full Changelog: v1.0.0...v1.0.1

Release v1.0.0

Choose a tag to compare

@milliondreams milliondreams released this 05 Apr 03:38
a4f2561

Uni 1.0

Uni was a graph database with vector search. As of 1.0, it's a reasoning engine.

Write rules. Attach probabilities to your edges. Uni propagates them through the graph — recursively, correctly — and gives you not just answers but confidence scores backed by full evidence chains.

Graph structure, vector similarity, logical inference, probabilistic reasoning. One embedded library. No server. pip install uni-db.


Probabilistic Locy

Locy already let you define recursive inference rules over your graph. Now those rules handle uncertainty.

MNOR combines independent risks — the probability that at least one of several causes fires. Three independent 30% failure modes give 65.7% combined risk, not 90%.

MPROD combines requirements — the probability that all conditions hold. Log-space arithmetic keeps it stable through long dependency chains.

Both converge correctly through recursive graph cycles.

Exact mode. When two proof paths share a base fact, independence is the wrong assumption. Enable exact_probability and Uni uses BDD-based weighted model counting for shared-evidence groups, fast path for everything else.

Provenance. Every derived fact carries its full proof trail — base facts, rules, paths, per-path probability. Request top_k_proofs and get the most probable derivation chains. When someone asks "why did the system flag this?" — you hand them the receipts.

Also new: PROB annotations, IS NOT complement semantics, strict_probability_domain mode, configurable convergence epsilon.

Three worked notebooks ship with 1.0: supply chain risk scoring, patent freedom-to-operate, regulatory impact cascades.


Stable API

We redesigned the public API around sessions and transactions — reads through sessions, writes through explicit transactions. Plus ~20 typed exceptions, cursor streaming, prepared statements, and a fluent schema builder. Modeled after rusqlite, Neo4j, and DuckDB.

uni_db.Database(path)    →  uni_db.UniBuilder.open(path).build()
db.query(...)            →  session = db.session(); session.query(...)
db.execute(...)          →  tx = session.tx(); tx.execute(...); tx.commit()
locy_evaluate(...)       →  session.locy(...)
except Exception         →  except uni_db.UniSchemaError / UniParseError / ...

Breaking change from 0.2 — and the last one we plan to make.


Correctness

Locy engine hardened. 9 runtime bugs found and fixed through the probabilistic work, including fold aggregation, recursive convergence, and projection correctness. 7,400 lines of new test scenarios. Locy TCK: 242/242. OpenCypher TCK: 3,896/3,897.

similar_to() now respects your distance metric. Was hardcoding cosine — L2 and dot product rankings were wrong. Fixed.

FTS sees unflushed writes. L0 buffer now merged into full-text results.

properties(n) returns current values after SET. L0 overlay applied in both schema and schemaless paths.


Under the Hood

  • Storage goes through a pluggable StorageBackend trait — Lance is the only backend, but the coupling is gone
  • Lance 1→3, Arrow 56→57, DataFusion 50→52
  • 56 Rust↔Python API gaps closed — full parity, type stubs grew from ~180 to ~985 lines

pip install uni-db · uni-db = "1.0.0" in Cargo.toml

[Docs](https://rustic-ai.github.io/uni-db) · [GitHub](https://github.com/rustic-ai/uni-db)

Built by [Dragonscale Industries](https://dragonscale.ai), part of [The Rustic Initiative](https://rustic.ai). Apache 2.0.

Full Changelog: v0.3.0...v1.0.0

Release v0.3.0

Choose a tag to compare

@github-actions github-actions released this 25 Mar 07:41
583c6b5

What's Changed

Other Changes

Full Changelog: v0.2.0...v0.3.0