Llm integration POC #1028

timfdev · 2025-08-12T22:23:17Z

pydantic-ai and ag-ui-protocol

need pydantic >= 2.10 and >=2.11.2 respectively, this breaks some of the unit tests

…s allowed token count. Make conflicting libraries pydantic-ai and ag-ui optional; disabling agent route if not installed. Make search routes async and fix small bugs in query building.

codspeed-hq · 2025-08-16T23:26:13Z

CodSpeed Performance Report

Merging #1028 will not alter performance

_{Comparing llm-integration (f837226) with main (98706ec)}

🎉 Hooray! `pytest-codspeed` just leveled up to 4.0.0!

A heads-up, this is a breaking change and it might affect your current performance baseline a bit. But here's the exciting part - it's packed with new, cool features and promises improved result stability 🥳!
Curious about what's new? Visit our releases page to delve into all the awesome details about this new version.

Summary

✅ 13 untouched benchmarks

…hestrator-core into llm-integration

codecov · 2025-08-18T15:53:58Z

Codecov Report

❌ Patch coverage is 44.62121% with 731 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.62%. Comparing base (98706ec) to head (f837226).

Files with missing lines	Patch %	Lines
orchestrator/search/indexing/indexer.py	23.78%	125 Missing ⚠️
orchestrator/search/indexing/traverse.py	32.52%	83 Missing ⚠️
orchestrator/search/agent/tools.py	7.81%	59 Missing ⚠️
orchestrator/search/retrieval/ranker.py	37.17%	49 Missing ⚠️
orchestrator/search/retrieval/utils.py	25.00%	42 Missing ⚠️
orchestrator/search/retrieval/validation.py	25.45%	41 Missing ⚠️
orchestrator/search/core/types.py	56.32%	38 Missing ⚠️
orchestrator/api/api_v1/endpoints/search.py	38.18%	34 Missing ⚠️
orchestrator/cli/search_explore.py	43.10%	32 Missing and 1 partial ⚠️
orchestrator/search/filters/date_filters.py	47.16%	28 Missing ⚠️
... and 14 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1028      +/-   ##
==========================================
- Coverage   84.93%   80.62%   -4.31%     
==========================================
  Files         214      245      +31     
  Lines       10408    11685    +1277     
  Branches     1020     1144     +124     
==========================================
+ Hits         8840     9421     +581     
- Misses       1295     2008     +713     
+ Partials      273      256      -17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

mrijk · 2025-08-19T14:48:44Z

orchestrator/api/api_v1/endpoints/search.py

+    db_model: Any,
+    response_schema: Type[BaseModel],
+    pk_column_name: str,
+    eager_loads: List[Any],


just out of curiosity: is there a reason that these 'old style' types are used? list vs List, etc.

Not really, just an old habit from me I guess. I see that the package enforces python >= 3.11, so I can take a look to remove all older style types, I think from 3.10 onwards alot of types I used here are not necessary anymore 👍

luc-tielen · 2025-08-19T09:15:47Z

orchestrator/api/api_v1/api.py

@@ -83,3 +84,5 @@
    tags=["Core", "Translations"],
 )
 api_router.include_router(ws.router, prefix="/ws", tags=["Core", "Events"])
+
+api_router.include_router(search.router)


tags=["Search"]? And should we add prefix = /search (and update the search router accordingly)?

Ah, I see you have it in the other file. Maybe that should be moved here to keep the "overview" of how routes are mounted in this file.

luc-tielen · 2025-08-19T09:41:05Z

orchestrator/cli/search_explore.py

+        dotenv run python main.py search semantic "Shop for an alligator store"
+        ...
+        {
+            "path": "subscription.shop.shop_description",
+            "value": "Kingswood reptiles shop"
+        },


We should make the examples more generic (also the ones below), since this is specific for the WFO instance where we built the initial POC.

luc-tielen · 2025-08-19T09:47:12Z

...migrations/versions/schema/2025-08-12_52b37b5b2714_search_index_model_for_llm_integration.py

+IDX_VALUE_TRGM = "ix_flat_value_trgm"
+IDX_CONTENT_HASH = "idx_ai_search_index_content_hash"
+
+TARGET_DIM = int(os.environ.get("EMBEDDING_DIMENSION", 1536))


Discussed this with you and Rene during standup, this migration probably needs to be turned into a script so it can be re-run when a different model is chosen.

We can do that but there needs to be a default vector field right. So once you run a second script to alter the column the migration will give conflicts.

I guess we can default it to 1536, the size of embeddings for openai text-embedding model(large).

Default 1536 👍

orchestrator/search/core/embedding.py

luc-tielen · 2025-08-19T20:30:41Z

orchestrator/search/docs/running_local_text_embedding_inference.md

+### Step back to the revision right before the dynamic-dimension migration
+
+```bash
+dotenv run python main.py db downgrade 850dccac3b02


I commented before that we needed a script for this, but does this mean you can run this migration independent from all other migrations?

Yes you can downgrade a specific migration and upgrade it again, essentially achieving the same as a cli script.

luc-tielen · 2025-08-19T20:36:37Z

orchestrator/search/filters/base.py

+FilterCondition = (
+    DateFilter  # DATETIME
+    | NumericFilter  # INT/FLOAT
+    | StringFilter  # STRING TODO: convert to hybrid search


do we need to make a ticket for this TODO?

Im thinking that maybe this stringfilter should be removed altogether, its already possible to do a hybrid search by passing a user query, passing something like the top 5 results back to the agent will probably yield better results.

For things like booleans/product blocks , we already have the equality filter. Matching on exact text by letting the agent fill in a string will probably not work well.

luc-tielen · 2025-08-19T21:17:14Z

orchestrator/search/retrieval/builder.py

+from orchestrator.search.schemas.parameters import BaseSearchParameters
+
+
+class QueryBuilder:


A bit nitpicky, but: this technically doesn't need to be a class, can be done with plain functions (you're not really using self).

timfdev added 8 commits August 13, 2025 00:20

Vector search and agent mode POC

99da746

l

960ce80

add ag-ui package

d34d467

fix linting

cede6f5

last lint fix

65e963d

Streaming pipeline for indexing, using litellm to track token count v…

90a5d1a

…s allowed token count. Make conflicting libraries pydantic-ai and ag-ui optional; disabling agent route if not installed. Make search routes async and fix small bugs in query building.

fix mypy issues & use pgvector image

d0a23ec

use pgvector for codspeed tests

4fce33d

timfdev and others added 3 commits August 17, 2025 01:30

Merge branch 'main' into llm-integration

b8b4eb8

update docs and cleanup

1ec3625

Merge branch 'llm-integration' of github.com:workfloworchestrator/orc…

5d4c316

…hestrator-core into llm-integration

mrijk reviewed Aug 19, 2025

View reviewed changes

use python 3.10+ style type hinting

f837226

luc-tielen reviewed Aug 19, 2025

View reviewed changes

		from orchestrator.search.schemas.parameters import BaseSearchParameters


		class QueryBuilder:

Llm integration POC #1028

Are you sure you want to change the base?

Llm integration POC #1028

Uh oh!

Conversation

timfdev commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging #1028 will not alter performance

🎉 Hooray! pytest-codspeed just leveled up to 4.0.0!

Summary

Uh oh!

codecov bot commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

timfdev commented Aug 12, 2025 •

edited

Loading

codspeed-hq bot commented Aug 16, 2025 •

edited

Loading

🎉 Hooray! `pytest-codspeed` just leveled up to 4.0.0!

codecov bot commented Aug 18, 2025 •

edited

Loading