[FEATURE] Add multimodal embedding support (image and video)

[Ndunge-Makau]

Scope check

• This is core LLM communication (not application logic)
• This benefits most users (not just my use case)
• This can't be solved in application code with current RubyLLM
• I read the Contributing Guide

Due diligence

• I searched existing issues
• I checked the documentation

What problem does this solve?

Currently, RubyLLM only supports text-based embeddings across all providers. However, Google's VertexAI offers multimodal embedding capabilities through their multimodalembedding model, which can generate embeddings for:

• Images (for visual search, similarity matching)
• Videos (for video content analysis)
• Combined text + image/video (for rich semantic understanding)

Users who need to:

• Build visual search systems
• Compare image/video similarity
• Create multimodal RAG (Retrieval-Augmented Generation) systems
• Generate embeddings for mixed media content

...are currently unable to leverage these capabilities through RubyLLM.

Proposed solution

Extend the existing Embedding.embed API to accept optional image and video parameters:

# Text + Image
RubyLLM.embed(
  "A red sports car",
  image: File.read('car.jpg'),
  model: 'multimodalembedding',
  provider: :vertexai
)

# Video with GCS URI
RubyLLM.embed(
  "Product demo video",
  video: 'gs://my-bucket/demo.mp4',
  model: 'multimodalembedding',
  provider: :vertexai
)

# Image-only (no text required)
RubyLLM.embed(
  image: image_data,
  model: 'multimodalembedding',
  provider: :vertexai
)

# Text only
RubyLLM.embed(
  "A blue sports car",
  model: 'multimodalembedding',
  provider: :vertexai
)

Implementation approach:

• Add image: and video: parameters to Provider#embed method
• Implement multimodal payload rendering in VertexAI::Embeddings module:
  • Support base64-encoded image data
  • Support video as base64 or GCS URIs (gs://...)
  • Handle optional text for pure image/video embeddings
• Standardize render_embedding_payload signature across all providers
• Return structured embeddings: { text: [...], image: [...], video: [...] }

Why this belongs in RubyLLM

• Feature parity: VertexAI already supports this; RubyLLM should be able to expose it.
• Unified API: Users expect all provider features through one interface
• Real demand: Visual search, RAG with images, content moderation all need this

I have a working implementation ready to submit as PR if there's interest!

Thanks for the review 😊

[fschwahn]

I'd have a use for this, it currently blocks us from migrating to ruby_llm.

Support video as base64 or GCS URIs (gs://...)

Seems weird to single out GCS? I guess it should work similar to the with option for the chat interface: https://rubyllm.com/chat/#working-with-images

[crmne]

Would be great to see a PR, but API design wise we need to keep the pattern established in the rest of the library: chat.ask("hello", with: ["file1.png", "file2.mp4"])

[kaka-ruto]

Thanks @crmne ! We will do that. @Ndunge-Makau has a working solution so she'll create a PR soon.

Right @fschwahn , GCS and other video options will use the same interface

[Ndunge-Makau]

Hey, @crmne, @fschwahn, @kaka-ruto,

Just submitted a PR for this. You can check it out here. Any feedback would be really appreciated.

Thanks 😊