Vector Aggregation function

[barryhunter]

Proposal:

It's probably a little unusual, but would be nice to be able to do query like

SELECT GROUPBY(), AVG(float_vector) FROM index WHERE ... GROUP BY year

ie get an 'average' vector from vector column. ie the average for all dimensions in the vector. In theory finds a 'central' vector for the group.
.... ie loops through all dimensions and calculates the average for each float.

In practice, it actually could just be SUM() that needed. as only the direction of the vector that important, can just sum up all the dimensions individually, and not bother dividing by the 'count'.
... would be ok to return unnormalized vector (can do that externally) - and because normalizing the actual magnitude will be changed anyway (normalizing will do the dividing)

Know it means processing a lot of data (ie going to have to sum up high dimensional vectors) but should still be better doing it internal to manticore, rather in the application.

Checklist:

^{To be completed by the assignee. Check off tasks that have been completed or are not applicable.}

• Implementation completed
• Tests developed
• Documentation updated
• Documentation reviewed
• OpenAPI YAML updated and issue created to rebuild clients

[tomatolog]

am I right that sum(vec) and avg(vec) should produce the vector with the same dimension as source vectors but not the single float?

[barryhunter]

YEs, the result would be a vector. In some crude way the resultant vector would capture the some semantic meaning of all the documents, in the group.

[sanikolaev]

@barryhunter What would you do next with those centroids?

[barryhunter]

Use KNN to find the document nearest the average vector. In theory it a good exampler for the cluster :)

But also averaging to compute a gridded dataset. For example, one of the primary datasets I'm searching is geolocated imagrying with CLIP embeddings, by grouping on a grid, then find the average vector for that 'cell'. Can build a new index of these grids, to find locations that match a particular vector.
Experimenting both as a location search, but as mapping the grid.

Also looking at using the resultant vector for zero-shot classification. eg have a small number of classifications, can then look for the nearest match, to come up with 'label' for the group. To sub-categorize the results.

[sanikolaev]

In theory it a good exampler for the cluster :)

Is there any study that confirms this (or maybe an example from another database)? I'm worried that vector embeddings in some groups (for example, "year" as in your original example) might be so different that their average won't make much sense. Even in other groups, like "electronics", I'm not sure the average embedding would really represent the concept well.

If you tried doing it manually and it worked, did it work well?

[barryhunter]

Ah, yes 'year' probably not the best example. Yes, in practice doing it on 'groups' that already have some semantic 'focus'.

My demo so far (i'll tidy it up so can put it online), is actually grouping by tags. ie taking images in a small area, and grouping in by image tag (so relatively similar images), I found that that in general using the 'average' vector produced a quite nice (although far from perfect!) example image. although I didn't use the vector if large numbers of images in the group, as you say they probably become very 'blurred' (would need to exclude outliers).

I'm not aware of any other system having the capacity. It may not be sensible thing to do :) Just something experimenting with.