Db sharding

[philippesaade-wmde]

The vector database architecture will be migrated to a sharded design based on language and entity type. Instead of a single database containing all vectors, the system will use a separate vector database per language per entity type.

Changes include:

• Separate each search implementation into its own file under /services/search.
• VectorSearch access changes from a single database to direct Astra collection access per database, with item/property collections split by language.
• HybridSearch now creates multiple VectorSearch instances (one per language), queries them in parallel, and merges results per entity.
• RRF is used to merge vector search and keyword search results in the search routes, while max similarity is used in the /similarity-score/ route.
• Query embeddings are reused across searches (vector search + keyword rescoring) instead of being recomputed independently.
• Similarity is now computed locally for better efficiency. This affects /similarity-score/ and keyword result similarity scoring.
• /similarity-score/ now supports both QIDs and PIDs, and validates that each result includes at least one of QID or PID.
• /similarity-score/ now enforces max 100 IDs.
• Item/property routes now cap K with MAX_VECTORDB_K (default 50) and enforce bounds during query validation.

[exowanderer]

Hello @philippesaade-wmde

I did my best to review the files. I still have about half to complete. Please feel free to add responses / comments on the 43 comments I left so far.

[exowanderer]

I was able to add a few more comments, mostly about if new functionality could break downstream (user) behaviour

[exowanderer]

I requested several descriptions per file in the "search" subdir

[exowanderer]

Added a few more requests for infomation

[exowanderer]

Will this break any downstream functionality for community contributors or WDEmbedding reusers?

[exowanderer]

Is this the original file that is now called HybridSearch.py?

[itamargiv]

Alright, I gave a few comments about accessibility and behavior, the client seems to look alright when I run it locally, but I'd recommend to still click around some more with the back-end on to see if there are any odd behaviors I missed (I only ran the vue dev server).

All in all it looks great, some nice improvements and clear user instructions.

[itamargiv]

Looks good on my part, thank you!