Skip to content

feat: implement global index registry for O(1) index discovery #198

@majiayu000

Description

@majiayu000

Summary

Currently, leann list scans directories to discover indexes, which can be slow for large directory trees. A global registry system would make index discovery O(1) by tracking all index paths centrally.

Proposed Solution

Implement a global registry at ~/.leann/indexes.json that stores the paths of all LEANN indexes:

{
  "indexes": [
    {
      "name": "my-docs",
      "path": "/Users/foo/projects/my-project/.leann/indexes/my-docs",
      "type": "cli",
      "created_at": "2024-12-25T10:00:00Z"
    },
    {
      "name": "app-index",
      "path": "/Users/foo/apps/data/app-index.leann",
      "type": "app",
      "created_at": "2024-12-25T11:00:00Z"
    }
  ]
}

Implementation Steps

  1. Register on build: When leann build creates an index, automatically register it in the global registry
  2. Register on API usage: When apps create indexes via the Python API, register them as well
  3. Update leann list: Read from registry instead of scanning directories
  4. Cleanup stale entries: Validate paths exist and remove stale entries periodically
  5. Migration: First run should scan existing indexes and populate the registry

Benefits

  • leann list becomes O(1) instead of O(n) directory scan
  • Works correctly even when running from $HOME or other large directories
  • Provides a single source of truth for all LEANN indexes

Related

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions