Add support for filtered semantic search

[ecairol]

Add support for filtered semantic search: filter layer, cache-key correctness, strategy heuristic, admin facets

Why

WPVDB\Search returns the nearest vectors for a query, with no way to constrain the result set. Integrators cannot ask for "posts in the XYZ category, published this year, by this author".

This is useful for folks that vectorize the full collection of categorized Posts, and want to implement a more granular search.

What changed

1. Filter layer (class-wpvdb-search.php)

Search::query() accepts a filters argument:

WPVDB\Search::query( array(
    'text'    => 'warm places',
    'filters' => array(
        'post_type'  => 'post',
        'tax_query'  => array( array( 'taxonomy' => 'category', 'field' => 'slug', 'terms' => array( 'beach' ) ) ),
        'date_query' => array( array( 'after' => '2024-01-01' ) ),
        'author'     => 7,
    ),
) );

Supported keys: post_type, post_status, author / author__in / author__not_in, post__in / post__not_in, date_query, tax_query, doc_type.

The filters apply in build_clauses(), which feeds both the native and the PHP fallback paths. The two engines therefore stay in agreement.

tax_query and date_query delegate to WP_Tax_Query::get_sql( 'e', 'doc_id' ) and WP_Date_Query. Core generates correct, escaped SQL against a non-wp_posts primary table, so this PR ships no query DSL of its own. Verified on WP 7.0.2.

No schema change. All facets derive from wp_posts and the term tables through doc_id.

2. Cache-key correctness

Cache previously keyed query results on version, text, model, and limit. A request filtered to one category would return results cached from an unfiltered request with the same text. That failure is silent and hard to reproduce, so it ships in the same PR as the filters.

Search::filters_cache_seed() canonicalizes the filter array and hashes it. REST folds the hash into the cache key. An empty filter set returns an empty seed, so existing cache keys stay byte-identical.

3. Strategy heuristic: pre/post filter.

A filtered vector search can run two ways. Pre-filtering scans the filtered subset exactly. Post-filtering uses the vector index across the entire dataset and discards non-matching rows. Neither wins in all cases.

Search::query() accepts strategy: auto, prefilter, or postfilter.

Condition	Strategy	Reason
candidates == 0	short-circuit	Returns before the embedding API call, so a doomed query costs nothing.
candidates <= 5000	pre-filter	Exact top-K over a small set, with no recall loss.
candidates / total >= 0.5	post-filter	The filter removes little, so the vector index earns its cost.
otherwise	pre-filter	Always correct.

Post-filtering widens its over-fetch window and retries while results come up short, bounded by wpvdb_search_max_topup_rounds.

Tuning filters: wpvdb_search_exact_scan_threshold, wpvdb_search_broad_ratio, wpvdb_search_overfetch_multiplier, wpvdb_search_max_topup_rounds, and wpvdb_search_strategy as a hard override.

The default is prefilter, not auto. See the open item below.

4. Admin facets and query plan

The Embeddings screen gains facet controls, built from core components (wp_dropdown_categories(), wp_dropdown_users(), submit_button()) inside the standard .tablenav filter row:

• Keywords, post type, category, tag on the main row.
• Author, date range, strategy override, and a query-plan toggle inside an Advanced search disclosure. It opens automatically when any advanced facet is active.

Controls submit as GET, so a tuned query is a shareable URL. Styles live in assets/css/admin.css and reflow at WP's 782px breakpoint.

The query-plan panel reports strategy and how it was chosen, candidates against total, engine and database type, top-up rounds, rows scanned, and timings. That panel is how the thresholds get calibrated against a real dataset.

It also populates the plan['candidates'] key, which was previously declared and never set.

Verification

Against MariaDB 11.7 with a live index:

• Every filter type matches ground truth counted directly from wp_posts and the term tables. cocktail returns 3 of 3, beer returns 0 of 0.
• prefilter and postfilter return identical result sets for the same query.
• The short-circuit path returns embed_ms=0 with an empty API key and raises no error, which confirms it skips the paid round trip.
• The top-up loop reaches round 2 when the first window misses the only match.
• Combined tax_query + post_type + date_query produces valid SQL on both query shapes.

11 new unit tests. The suite passes at 224 tests and 824 assertions. PHPCS reports no new violations.

Follow-ups

• Replicate on REST args schema and WP-CLI
• Remove the dead meta column scaffolding

[ecairol]

@rbcorrales you have a better understanding of how this could play out with the other plugins like Search Blocks and Smart Search.

I think it's useful to have the filter options at the core WPVDB level, so that's what I opened this PR, but let me know your thoughts.

The main use case I see is for blogs where there are many Post Categories and they want to (for instance) implement a Chatbot that feeds from only 1 of the existing Categories. For example, a Travel site might have Posts about places, tours and restaurants, but also posts about Gift Guides and Style, and they might want to filter out the latter for a "Travel Agent" bot (this is a real life example).

I noticed there' a meta column in the vectorized data, but it's not being used and is a LONGTEXT, so it wouldn't store complex relationships easy. I also think it would be repetitive if we already have categorized data in postmeta. Alternatively, we can build an embeddings_meta table later if we see it fits.

With this changes, the centralized SQL query now receives the WP taxonomies and other metadata as optional arguments.

[Copilot]

Pull request overview

This PR extends WPVDB’s semantic search to support WordPress-style filtering (post type/status, author constraints, tax/date queries, and doc_type), and integrates the filter set into cache keys to prevent cross-contamination between filtered and unfiltered cached results. It also introduces a prefilter/postfilter strategy mechanism (with an “auto” heuristic) and adds admin UI facets plus a query-plan panel on the Embeddings screen to tune and observe filtered searches.

Changes:

• Add a filter layer to Search::query() / Search::build_clauses() plus canonicalization + hashing for stable cache keys.
• Add strategy selection (prefilter/postfilter/auto) including a postfilter top-up loop and candidate counting.
• Add Embeddings admin search facets (GET-based) and query-plan display, with supporting CSS.

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/unit/SearchTest.php	Adds unit coverage for filter canonicalization, cache seed stability, clause building behavior, and strategy normalization.
includes/class-wpvdb-search.php	Implements filters, candidate counting, strategy heuristic, postfilter execution, and cache seeding utilities.
includes/class-wpvdb-rest.php	Incorporates filters into REST query execution and into the cache key override to avoid incorrect cache hits.
includes/class-wpvdb-query.php	Allows WP_Query integration to pass vdb_filters into the vector search layer.
assets/css/admin.css	Adds responsive layout styles for the new Embeddings facet controls.
admin/views/embeddings.php	Adds Embeddings search facets, strategy override control, and query-plan reporting UI.
.gitignore	Ignores a new local cache directory under docker setup.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 6 out of 7 changed files in this pull request and generated no new comments.

Suppressed comments (3)

includes/class-wpvdb-search.php:755

• Using GROUP BY e.id while selecting multiple non-aggregated columns ({$columns} plus computed distance) can fail on MySQL/MariaDB when ONLY_FULL_GROUP_BY is enabled (SQL error about non-grouped columns). To de-duplicate rows introduced by taxonomy joins without relying on relaxed SQL modes, prefer SELECT DISTINCT and drop the GROUP BY.

			GROUP BY e.id

includes/class-wpvdb-search.php:819

• Same GROUP BY e.id issue as the native path: selecting many non-aggregated columns with GROUP BY can break under ONLY_FULL_GROUP_BY. If the goal is to de-duplicate rows caused by joins, SELECT DISTINCT is safer and portable.

				GROUP BY e.id

includes/class-wpvdb-search.php:368

• When tax_query adds joins, this query can return duplicate e.id values. Since the result is only used as a set membership test, selecting distinct IDs avoids unnecessary duplicates and reduces work in PHP when building $allowed.

			"SELECT e.id FROM {$table} e{$clauses['join']} WHERE {$where}",