Skip to content

Benchmark WaveMind against real LoCoMo memory systems - #66

Merged
CaspianG merged 2 commits into
mainfrom
goal4/public-memory-competitors
Jul 28, 2026
Merged

Benchmark WaveMind against real LoCoMo memory systems#66
CaspianG merged 2 commits into
mainfrom
goal4/public-memory-competitors

Conversation

@CaspianG

Copy link
Copy Markdown
Owner

Summary

  • add real Mem0 OSS 2.0.14 and Hindsight OSS 0.8.5 adapters to the shared LoCoMo evidence protocol
  • publish exact-commit results for 5,882 memories and 1,977 evidence queries alongside WaveMind, Memory OS, Chroma, and Qdrant
  • expose query latency and ingest scope separately, require native-version/profile/provenance metadata, and prevent local dataset paths from entering public artifacts
  • refresh the benchmark registry, reports, leaderboard, dashboard, and README links

Measured result

Engine Recall@5 Precision@1 MRR@5 Avg query
WaveMind 0.548 0.333 0.432 4.88 ms
WaveMind + Memory OS 0.548 0.332 0.431 5.99 ms
Mem0 OSS 0.500 0.263 0.369 270.24 ms
Chroma static 0.408 0.219 0.305 4.12 ms
Qdrant static 0.409 0.219 0.305 103.27 ms
Hindsight OSS 0.316 0.052 0.148 320.61 ms

This is retrieval evidence on one local machine, not final answer quality or hosted-service throughput. Internal systems start ingest from shared precomputed embeddings; Mem0 and Hindsight ingest includes their native embedding and persistence work.

Validation

  • python -m pytest -q -> 887 passed, 5 skipped
  • python -m ruff check wavemind tests benchmarks examples -> passed
  • benchmark artifact audit -> pass (45 implemented, 3 runner-ready, 5 planned)
  • python -m build -> passed
  • python -m twine check dist\\* -> passed
  • public evidence scan -> no secrets or local absolute paths

@CaspianG
CaspianG merged commit 1f5940d into main Jul 28, 2026
21 checks passed
@CaspianG
CaspianG deleted the goal4/public-memory-competitors branch July 28, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant