Questions/discussions about this project

[stueble]

Hi Henning,

while starting to implement a searchless-extension for paperless-ngx, I found your searchless-ngx github project... :-) Are you planning to continue development on this project? I would like to discuss a few topics.

I am currently using OpenAPI, but this should not make a big difference I assume:

• My main focus is to use paperless-gpt to add more structures meta information, such as referenced year, referenced month, amount, consumption, consumption_unit, etc. My expectation is that this way, many questions can be answered more correctly and by only accessing the custom_values.
• In my prototype, I use an interface allowing the LLM to query the available metadata (including custom_fields) before starting the document query, hoping that it creates better search queries such as search_document( q="", type="Rechnung", tag="Strom", bezugsjahr=2025 ) if you ask "What have I spent in 2025 for electricity". Have you any experiences in this direction?
• Wouldn't it make sense to extract the semantic search part into a separate module?

Best regards from cologne,
Chris

[hensing]

Hi Chris,

Sorry for the late response and great to hear from you, and nice that searchless-ngx caught your eye! Short answer: yes, I'm actively continuing development — happy to discuss any of this.

Greetings from Bonn/Cologne too, by the way. :-)

A few thoughts on your points:

OpenAI / providers: No problem at all. I recently refactored the provider layer to be pluggable
— you can pick the embedding and chat backend independently (Mistral, Google, OpenAI, or
self-hosted via Ollama), so OpenAI is fully supported. So we're on the same page there.

Structured metadata via paperless-gpt (year, amount, consumption, …): This matches my own
experience and is exactly the direction I believe in. For quantitative/factual questions ("how
much did I spend on electricity in 2025?"), pure vector search is the wrong tool — you want
deterministic answers from structured fields, not fuzzy semantic matches. searchless-ngx already
indexes custom fields into the vector store's metadata (as cf_*), and there's a real
complementarity here: paperless-gpt writes the structured fields, searchless-ngx (read-only by
design) queries them.

LLM querying available metadata before searching: You've independently arrived at the core
design principle of the project: searchless-ngx exposes exactly this pattern via MCP tools — the
LLM first discovers the available correspondents, tags, document types and custom fields
(there's a cached metadata layer and a master-data tool with fuzzy name matching for
typos/abbreviations like "DB" → "Deutsche Bahn"), and then builds a precise filtered query. So
your search_document(q="", type="Rechnung", tag="Strom", bezugsjahr=2025) intuition is precisely
the goal. One honest caveat: today custom fields are indexed but not yet usable as direct
filter dimensions in the semantic-search tool — that's high on my roadmap, and your use case is
a perfect motivator to prioritize it.

Separating the semantic-search part into its own module: Agreed in principle — it's already a
fairly self-contained semantic/ package (vector store, chunking/sync, metadata cache) decoupled
from the MCP/server and Paperless-API layers. Spinning it out into a standalone module/service
is a sensible next step, and worth designing together so it fits both our setups.

I'd genuinely enjoy talking these through - happy to catch up soon!

Best regards,
Henning