Python: Feature python vector stores preb

[eavanvalkenburg]

Motivation and Context

Overhaul of the VectorStores to get it matching with dotnet and the latest development there.

Closes:

• Port dotnet feature: .Net: Delete CreateCollection, rename CreateIfNotExists and make EnsureExists thread safe. #11938
• Port dotnet feature: .Net: Rename IsFilterable to IsIndexed and IsFullTextSearchable to IsFullTextIndexed #11598
• Port dotnet feature: .Net: Add delete and exists methods for IVectorStore #11597
• Port dotnet feature: .Net: Refactor MEVD Vector Attribute and make Dimensions mandatory #11517
• Python: Implement Hybrid Search for all memory connectors #10561
• Python: Support BetweenFilterClause or RangeFilterClause for Redis #10391
• .Net: Python: Support listing all keys on vector store #9911
• .Net: Python: API for listing/filtering records without similarity search #9892
• Python: Implement new filters #10867

Description

Many changes, some highlights:

• Moved from different vector fields types, to a single with a enum first VectorStoreField()
• Renamed several items
• Made embeddinggenerrator a part of the vector field spec and do automatic vectorization from there or from the same param on a collection
• Adds hybrid search using the same setup
• Removed the intermediate TextSeachVectorSearch object need, by allowing create_search_function directly on a collection

Will write out full changes in a migration guide in the docs.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
__init__.py	4	0	100%
const.py	7	0	100%
kernel_types.py	10	0	100%
connectors
_search_shared.py	28	0	100%
azure_ai_search.py	303	27	91%	159–160, 208, 210–213, 215, 244, 246, 248, 339, 361–362, 423, 425, 439–440, 451, 522, 544, 554, 566, 586–587, 781–782
azure_cosmos_db.py	401	166	58%	184, 201, 207–209, 319, 321, 396, 400, 416–419, 421–424, 426–427, 440–443, 446–448, 454–455, 457, 464–466, 470–472, 495–497, 503–505, 511–512, 514, 519, 536, 624, 627–628, 636, 646–650, 705, 708–709, 745, 747, 760–761, 765–767, 778–781, 784–785, 787–788, 793–797, 800–803, 805–810, 812–813, 820–825, 844–846, 848–851, 859, 866–871, 873–887, 889–900, 903, 905–906, 909–910, 913, 915–920, 926, 930, 954, 982–983, 987–992, 1081–1086
brave.py	142	23	83%	28, 171, 177, 187, 206–209, 245, 250, 252, 270–281
chroma.py	255	89	65%	104, 107–111, 126–127, 135–136, 165, 167, 175, 187–190, 197, 216, 220, 241, 249, 253, 266, 270, 275–276, 285–288, 304–305, 313, 332, 343, 345, 349, 367–369, 371–386, 388–398, 400–406, 408, 410, 412–413, 416–417, 419–420, 423–425, 447–448, 450
faiss.py	119	12	89%	48, 58–59, 63–64, 121, 129, 133, 170, 217, 220, 240
google_search.py	145	21	85%	29, 217, 227, 249–254, 269–270, 303–312
in_memory.py	169	23	86%	69–70, 74, 78–81, 116, 128, 130, 171, 175, 179, 194, 211, 226, 266–267, 269, 276–277, 282, 317
memory.py	10	10	0%	3, 5, 48–53, 56–57
mongodb.py	258	128	50%	100–101, 105, 117–120, 125–126, 133–134, 139–140, 149, 200, 254, 282, 284, 294–296, 302–304, 311, 315, 327–328, 336, 347–351, 360–363, 366–368, 374–375, 377, 384–386, 390–392, 404–407, 410–412, 418–419, 421, 428–430, 434–436, 444–446, 448–463, 465–475, 477–483, 485–490, 492, 494–495, 498–499, 501–502, 505–507, 512, 516, 577–578, 580, 622–623, 627–628
pinecone.py	333	79	76%	130, 144–147, 152, 157, 183, 216, 220–221, 228, 250, 254, 282, 310–313, 329, 375, 377, 382, 389, 400, 402, 404, 406, 409, 418, 420, 424, 439, 441, 443, 448, 455, 467, 474, 476, 501–507, 513, 515, 519–528, 531–536, 539–541, 543, 549–551, 556, 561, 567–568, 571, 595, 664
postgres.py	371	101	72%	139, 157, 184, 297–298, 383, 389–391, 396–397, 399–400, 407, 429, 485, 487, 489, 506, 518, 568, 577, 627, 647, 667, 672, 678, 690, 720, 724, 726, 748, 775, 780, 797, 854–856, 858–884, 886–892, 894–899, 901, 903–904, 907–908, 910–911, 914–923, 925, 930, 934, 961–963
qdrant.py	251	57	77%	249, 251, 287, 306–310, 316–317, 320–321, 327–330, 332, 334, 351, 375–381, 388, 391, 394–403, 406–411, 414–418, 423, 429–430, 433, 436–437, 495, 509
redis.py	414	170	58%	102, 107, 120, 123, 127–129, 133–134, 137, 147–151, 164–165, 299, 304–305, 316–321, 324–325, 333–335, 337, 344–348, 350, 352–354, 363, 369, 371, 375–385, 387–389, 391–401, 403–423, 425, 427–446, 449–461, 463–468, 470, 472–473, 476–477, 479–480, 483–485, 501–506, 510, 566, 568, 631–638, 694, 696, 747–754, 837, 844–845
search.py	10	10	0%	3, 5, 19–24, 27–28
sql_server.py	501	88	82%	152, 349–351, 371, 390, 401, 403, 426, 472–475, 491, 503–506, 528, 530, 574–580, 586–587, 589–590, 594–608, 611–616, 619–623, 628, 634–635, 638, 643, 645–648, 787, 806–808, 814, 820, 825, 830–833, 839, 873, 915, 922–924, 1090–1091, 1093, 1095, 1116
weaviate.py	335	142	57%	110, 315–318, 327–328, 330–332, 337, 341–342, 353–355, 364–372, 375–376, 380–381, 384–386, 389–390, 395–396, 399–400, 402, 411–413, 415–424, 426, 428–443, 447–453, 455, 457, 459–460, 463–464, 466–467, 470–472, 482–483, 486–487, 493–494, 497–498, 502, 505–509, 524, 539, 548–549, 563, 567–570, 573–574, 580–581, 585, 588–589, 592, 595, 615, 634, 650, 659–660, 665–666, 671, 765, 776–781, 786–791, 796–797
connectors/ai/embeddings
embedding_generator_base.py	10	1	90%	9
connectors/ai/hugging_face
hf_prompt_execution_settings.py	24	1	95%	33
connectors/ai/hugging_face/services
hf_text_embedding.py	32	5	84%	84–88
connectors/ai/open_ai/prompt_execution_settings
azure_chat_prompt_execution_settings.py	73	0	100%
core_plugins
text_memory_plugin.py	33	8	75%	14, 43, 72, 78–80, 82, 103
data
__init__.py	0	0	100%
_shared.py	62	9	85%	136, 142, 168, 172–173, 175, 178–179, 181
text_search.py	69	4	94%	229, 240, 277, 279
vector.py	667	118	82%	83, 413, 579, 698, 896–899, 905, 907–908, 1026, 1064–1065, 1068, 1075, 1095–1102, 1104, 1106–1107, 1109, 1111, 1114–1119, 1121–1126, 1148, 1171, 1251–1253, 1333, 1484, 1499, 1505–1506, 1508, 1535, 1609–1610, 1622–1623, 1652, 1737, 1742–1745, 1859, 1886–1887, 1926, 1946, 1954–1955, 1967, 1974, 2019, 2022, 2076–2078, 2081, 2090, 2113, 2115–2120, 2122–2127, 2131–2137, 2141–2146, 2149–2151, 2153, 2181, 2192, 2206, 2214, 2253, 2277, 2291, 2337, 2377
exceptions
vector_store_exceptions.py	24	0	100%
functions
kernel_plugin.py	174	3	98%	395, 398, 411
TOTAL	26293	4445	83%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3600	18 💤	0 ❌	0 🔥	1m 52s ⏱️

[westey-m]

We actually removed this method, and only have ensure_collection_exists now.

[westey-m]

FYI, we don't have something like this on the c# side.

[eavanvalkenburg]

Really? I got it from there... It's it not in mevd?

[westey-m]

We only have EnsureCollectionCreated and EnsureCollectionDeleted:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Connectors/VectorData.Abstractions/VectorStorage/VectorStoreCollection.cs

[westey-m]

FYI, we are just calling this collection_exists on the c# side.

[westey-m]

FYI, we have this as collection_exists on the c# side.

[westey-m]

FYI, we are calling this VectorSearchable on the c# side.