improve configurability of embedding and LLM model sources

[ChuckHend]

Issue is WIP and will be further refined.

LLM and embedding model sources are currently defined in GUC, e.g. vectorize.openai_service_url = https://api.openai.com/v1 contains the base url for OpenAI. This implementation introduces at least two limitatations:

1. two vectorize projects cannot use two different urls for the same source. for example, project_a wants vectorize.openai_service_url = https://api.openai.com/v1 and project_b wants vectorize.openai_service_url = https://myapi.mydomain.com/v1. The pg vectorize background worker reads the model source from the job's values in the vectorize.job table in order to know which guc to use.
2. adding a new embedding source is cumbersome. adding GUCs requires a code change, and a new project relase.

Proposal: move GUCs into a table such as vectorize.model_sources to contain information such as the base url, schema, etc.

Considerations:

• How to handle API keys for each record in model source. Can this be a superuser only table, or does API key need to be a GUC?
• API request/response schemas. These are currently defined in code, but we could also define these in json mapping using a framework like jolt . There is an experiment of this implementation here. Using this framework would allow for adding arbitrary embedding and LLM sources via inserts to this model sources table.

[tavallaie]

what is different between a project and a table?
how about using multiple api if they have same embedding model?
can we have different embedding columns? do you think it can be useful or not? like embedding for different language (like Arabic ,Persian, English etc ) or having image embedding along side of texts?

[ChuckHend]

what is different between a project and a table?

I was using "project" and "job_name" (the parameter in vectorize.table()) interchangeably. Maybe we should rename it to "project". A table can have multiple of the "jobs". Tables have column(s) that get transformed into embeddings using the transformer specified in the vectorize.table() call. So if you change any of the parameters, it should be a new job.

how about using multiple api if they have same embedding model?

this is currently possible and we'd want to preserve it going forward. Currently the model provider is determined by the name of the model passed into the function call, and it works the same for embeddings and LLM. For example, ollama/wizardlm2:7b will route the request to the server at vectorize.ollama_service_url and openai/text-embedding-ada-002 will route to vectorize.openai_service_url. So theoretically if you could have ollama/mymodel and openai/mymodal and they should both work assuming the models actually exist at the destination servers.

can we have different embedding columns? do you think it can be useful or not? like embedding for different language (like Arabic ,Persian, English etc )

I think we could do something like the below for multiple language models.

vectorize.table(
  table => 'mytable',
  job_name => 'project_persian',
  transformer => 'ollama/persian-model'
);
vectorize.table(
  table => 'mytable',
  job_name => 'project_arabic',
  transformer => 'ollama/arabic-model'
);

having image embedding along side of texts

Images are not yet supported but there are plans to implement it soon.

[tavallaie]

I suggest building a simple secret manager using Postgres with tools like pgcrypto or pgsodium
. The goal is to securely store sensitive information like API keys.

Here’s the plan:

1. Use key-value storage for each secret, with an expiration option to set time limits if needed.

2. Optionally, we could add features like rate limits in the future, but we can leave this out for now to keep things simple.

The table structure would store user-level secrets, so it doesn’t have to be a super-user table. This way, each user can securely store and manage their own API keys.

Also, since transformer and chat_model are similar (they’re the same model but respond to different requests), we could set up a single table called model_resource and use a type field to separate them.

What do you think, @ChuckHend?

[ChuckHend]

I like this. What would a row look like for OpenAI since base url and api key would be for both an LLM type and embeddings type? Whereas some others might be just embedding, or just LLM model_resource?

[tavallaie]

we have some difficulty here, some LLMs limits their responsible embeddings with what they support, so I need to think about it.

[tavallaie]

I think there are two ways to do this:

First Way: Using compatible_models as a Column

We can add a compatible_models column to the model_resource table:

Column Name	Data Type	Description
model_id	UUID	Primary key
name	VARCHAR	Model name (e.g., 'openai/text-embedding-ada-002')
type	VARCHAR[]	Types of the model (e.g., ['embedding', 'LLM'])
base_url	VARCHAR	Base URL for API requests like 'https://api.openai.com/v1/embeddings'
capabilities	JSONB	Details on languages, modalities, etc.
request_template	JSONB	JSON template for constructing API requests
response_mapping	JSONB	JSON mapping for parsing API responses
compatible_models	UUID[]	Array of model_ids that are compatible

the second way is making another table for model_compatibility instead of compatible_models column:

Column Name	Data Type	Description
model_id	UUID	Foreign key to model_resource (model)
compatible_model_id	UUID	Foreign key to model_resource (compatible)

[ChuckHend]

@tavallaie, what are some example of values that would go in the compatible_models column?

[tavallaie]

In my design, there is no difference between embedding models and LLMs or even images and audio.
So if there is any dependent model like embedding for LLMs, we put their UUID or any other unique identifier in that column.

[ChuckHend]

Ok I think I might see where you are going with that. Can you provide an example of what that table might look like in your use case?

[tavallaie]

I am thinking of something like this:

model_id	name	type	base_url	capabilities	request_template	response_mapping	compatible_models
'1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d'	'openai/text-embedding-ada-002'	['embedding']	'https://api.openai.com/v1/embeddings'	{ "languages": ["en"], "modalities": ["text"] }	{ "model": "text-embedding-ada-002", "input": "{{input_text}}" }	{ "embedding": "$.data[0].embedding" }	['3d4c5b6a-7e8f-9a0b-1c2d-3e4f5a6b7c8d']
'3d4c5b6a-7e8f-9a0b-1c2d-3e4f5a6b7c8d'	'custom/advanced-LLM'	['LLM']	'https://api.custom.com/v1/generate'	{ "languages": ["en", "es"], "modalities": ["text"] }	{ "model": "advanced-LLM", "prompt": "{{prompt_text}}" }	{ "generated_text": "$.choices[0].text" }	['1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d']

[ChuckHend]

Thank you. Do the APIs need to change how they reference a model then, or how does this impact vectorize.table(), vectorize.init_rag(), and others?