Skip to content

fix: count stored rows, not sealed row_count, in search and stats (#534) - #745

Open
DrTrippel wants to merge 1 commit into
zilliztech:mainfrom
DrTrippel:fix/534-count-stored-rows
Open

fix: count stored rows, not sealed row_count, in search and stats (#534)#745
DrTrippel wants to merge 1 commit into
zilliztech:mainfrom
DrTrippel:fix/534-count-stored-rows

Conversation

@DrTrippel

Copy link
Copy Markdown
Contributor

Fixes #534

Problem

On Milvus Server, memsearch search and memsearch stats report a collection as empty for about 10 minutes after memsearch index succeeds. The data is not lost. memsearch just doesn't see it.

MilvusStore.search() guards against the #306 BM25 crash by returning [] when get_collection_stats()["row_count"] == 0, and count() returns the same value. row_count only counts sealed/flushed segments. Freshly upserted rows sit in a growing segment until the DataNode syncs them (dataNode.segment.syncPeriod, 600s by default). Until then:

  • row_count is 0, so search returns nothing without querying;
  • stats prints Total indexed chunks: 0.

Collections that already have an older flushed segment are not affected: the guard passes, and hybrid_search sees the growing rows immediately. That is why the bug only shows up on new or freshly re-created collections.

Milvus Lite flushes eagerly, which masks the problem, so #534 looked like a missing flush.

Repro (Milvus v3.0.1 standalone in Docker, memsearch 0.4.20, pymilvus 3.0.1)

Step Result
memsearch index <dir> --collection ms_probe Indexed 1 chunks.
get_collection_stats("ms_probe")["row_count"] 0
query("ms_probe", filter="", output_fields=["count(*)"], consistency_level="Strong") 1
memsearch search <canary> --collection ms_probe No results found. until 608s after indexing, then found
Existing 23-row collection, stats / search 0 / nothing; after MilvusClient.flush(...), 23 and a score-1.0 hit

Fix

Tests

New tests

Local runs

  • uv run python -m pytest tests/test_store.py: 19 passed.
  • uv run python -m pytest: 423 passed, 7 skipped, 6 failed. The same 6 fail on unmodified main (test_claude_stop_indexes_only_server[*], test_session_start_and_data_commands_are_scoped_to_host_project[*]), so they are unrelated to this change.
  • ruff check and ruff format --check: clean.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Ng6tyJJrBWCrSZNKGUbBdc

…lliztech#534)

get_collection_stats()["row_count"] only counts sealed/flushed segments. On
Milvus Server, freshly upserted rows stay in a growing segment until the
DataNode syncs them (syncPeriod, 600s by default), so row_count reads 0.
MilvusStore.search() returned [] on row_count == 0 (the zilliztech#306 empty-collection
guard) and count() reported the same number, so a new collection looked
empty to `memsearch search` and `memsearch stats` for ~10 minutes after
`memsearch index` succeeded.

count() now uses a strong-consistency count(*) query, and search() guards on
count() == 0, which keeps the zilliztech#306 protection for truly empty collections.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ng6tyJJrBWCrSZNKGUbBdc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MilvusStore.upsert() reports success but writes are not durable on remote Milvus 2.5+ (missing flush)

1 participant