[Bug]: GoogleBatchEmbeddings.batch_embeddings() doesn't include auth_header when using custom api_base for Gemini embeddings

[qdrddr]

What happened?

A bug happened!

Bug Summary

When using the LiteLLM SDK's embedding() function with Gemini models and a custom api_base parameter (e.g., for Cloudflare AI Gateway routing), the authentication header is not included in the HTTP request, causing authentication failures.

================================================================================
BUG #1: Missing Authorization Header (x-goog-api-key)

EXPECTED BEHAVIOR:
When api_key is provided, request should include:
headers = {
'Content-Type': 'application/json; charset=utf-8',
'x-goog-api-key': 'AIzaSyAS0bSJ2TaEvFte...' ← SHOULD BE SENT
}

ACTUAL BEHAVIOR:
Request only includes:
headers = {
'Content-Type': 'application/json; charset=utf-8'
# ← x-goog-api-key is MISSING!
}

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.

✅ BUG CONFIRMED: Request failed with authentication error
Error: litellm.BadRequestError: GeminiException BadRequestError - {
"error": {
"code": 403,
"message": "Method doesn't allow unregistered callers (...

================================================================================
BUG #2: Ignored Extra Headers Parameter

EXPECTED BEHAVIOR:
When extra_headers is provided, request should include:
headers = {
'Content-Type': 'application/json; charset=utf-8',
'X-Custom-Header': 'test-value', ← SHOULD BE SENT
'Authorization': 'Bearer token' ← SHOULD BE SENT
}

ACTUAL BEHAVIOR:
extra_headers parameter is completely ignored
Request only includes Content-Type header

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.

✅ BUG CONFIRMED: extra_headers were ignored
If extra_headers worked, Authorization header would be sent
Error: litellm.BadRequestError: GeminiException BadRequestError - {
"error": {
"code": 403,
"message": "Method doesn't allow unregistered callers (...

================================================================================
ROOT CAUSE

File: litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
Method: GoogleBatchEmbeddings.batch_embeddings()

Lines 60-62 create headers WITHOUT including auth_header:
headers = {'Content-Type': 'application/json; charset=utf-8'}

← auth_header is retrieved but NEVER added!

← extra_headers parameter is NEVER checked!

================================================================================
FIX (2 lines)

headers = {'Content-Type': 'application/json; charset=utf-8'}
if auth_header:
headers.update(auth_header)

Environment

• LiteLLM Version: 1.80.10 (latest as of 2025-12-15)
• Python Version: 3.13
• Provider: Google Gemini (gemini/gemini-embedding-001)
• Use Case: Routing through Cloudflare AI Gateway (CFAIG)

Steps to Reproduce

from litellm import embedding
import os

# Cloudflare AI Gateway URL (noauth mode)
cfaig_url = "https://gateway.ai.cloudflare.com/v1/{account_id}/noauth/google-ai-studio/v1beta"

response = embedding(
    model="gemini/gemini-embedding-001",
    input=["What is the meaning of life?"],
    api_key=os.getenv("GEMINI_API_KEY"),
    api_base=cfaig_url,
)

Expected Behavior

The request should include the x-goog-api-key header with the Gemini API key, similar to how the completion() function handles Gemini models with custom api_base.

Actual Behavior

The request is sent without any authentication header, resulting in:

httpx.HTTPStatusError: Client error '401 Unauthorized'
litellm.exceptions.AuthenticationError: GeminiException - {"success":false,"result":[],"messages":[],"error":[{"code":2009,"message":"Unauthorized"}]}

Root Cause

File: litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py

Method: GoogleBatchEmbeddings.batch_embeddings()

The bug is in the batch_embeddings() method:

1. Line 26-38: The _get_token_and_url() method correctly retrieves auth_header containing the authentication credentials
2. Line 60-62: Creates a headers dict but never includes the auth_header
3. Line 87-91: Sends the HTTP request with incomplete headers (missing authentication)

Problematic Code:

# Line 26-38: Gets auth_header correctly
auth_header, url = self._get_token_and_url(
    model=model,
    auth_header=_auth_header,
    gemini_api_key=api_key,
    ...
    api_base=api_base,
    ...
)

# Line 60-62: Creates headers WITHOUT including auth_header ❌
headers = {
    "Content-Type": "application/json; charset=utf-8",
}
# ❌ BUG: auth_header is never added to headers!

# Line 87-91: Sends request with incomplete headers
response = sync_handler.post(
    url=url,
    headers=headers,  # Missing authentication!
    data=json.dumps(request_data),
)

Proposed Fix

Add the auth_header to the headers dict before sending the request:

# Line 60-66: Include auth_header in headers ✅
headers = {
    "Content-Type": "application/json; charset=utf-8",
}

# ✅ FIX: Add auth_header if it exists
if auth_header:
    headers.update(auth_header)

Additional Context

• The completion() function handles this correctly for Gemini models with custom api_base
• The _check_custom_proxy() method (line 46) correctly sets auth_header = {"x-goog-api-key": gemini_api_key} when api_base is provided
• This bug affects all use cases where Gemini embeddings are routed through a proxy/gateway with custom api_base

Impact

This bug prevents users from:

• Routing Gemini embeddings through Cloudflare AI Gateway for caching/analytics
• Using any custom proxy/gateway for Gemini embeddings
• Benefiting from edge caching and cost optimization features

Related Issues

This appears to be related to the fix in PR #16085 which fixed direct Gemini API access, but the fix didn't cover the case when api_base is set.

Relevant log output

================================================================================
BUG #1: Missing Authorization Header (x-goog-api-key)
================================================================================

EXPECTED BEHAVIOR:
  When api_key is provided, request should include:
    headers = {
      'Content-Type': 'application/json; charset=utf-8',
      'x-goog-api-key': 'AIzaSyAS0bSJ2TaEvFte...'  ← SHOULD BE SENT
    }

ACTUAL BEHAVIOR:
  Request only includes:
    headers = {
      'Content-Type': 'application/json; charset=utf-8'
      # ← x-goog-api-key is MISSING!
    }


Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.

✅ BUG CONFIRMED: Request failed with authentication error
   Error: litellm.BadRequestError: GeminiException BadRequestError - {
  "error": {
    "code": 403,
    "message": "Method doesn't allow unregistered callers (...

================================================================================
BUG #2: Ignored Extra Headers Parameter
================================================================================

EXPECTED BEHAVIOR:
  When extra_headers is provided, request should include:
    headers = {
      'Content-Type': 'application/json; charset=utf-8',
      'X-Custom-Header': 'test-value',  ← SHOULD BE SENT
      'Authorization': 'Bearer token'   ← SHOULD BE SENT
    }

ACTUAL BEHAVIOR:
  extra_headers parameter is completely ignored
  Request only includes Content-Type header


Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.

✅ BUG CONFIRMED: extra_headers were ignored
   If extra_headers worked, Authorization header would be sent
   Error: litellm.BadRequestError: GeminiException BadRequestError - {
  "error": {
    "code": 403,
    "message": "Method doesn't allow unregistered callers (...

================================================================================
ROOT CAUSE
================================================================================
File: litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
Method: GoogleBatchEmbeddings.batch_embeddings()

Lines 60-62 create headers WITHOUT including auth_header:
  headers = {'Content-Type': 'application/json; charset=utf-8'}
  # ← auth_header is retrieved but NEVER added!
  # ← extra_headers parameter is NEVER checked!

================================================================================
FIX (2 lines)
================================================================================
headers = {'Content-Type': 'application/json; charset=utf-8'}
if auth_header:
    headers.update(auth_header)
================================================================================

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.80.10

Twitter / LinkedIn details

No response

[qdrddr]

LiteLLM Gemini Embedding Bugs - Confirmation Log

Test Date

2025-12-15

Environment

• LiteLLM Version: 1.80.10
• Python Version: 3.13
• Test Script: ai/tests/confirm_bug.py

Bugs Demonstrated

Bug #1: Missing Authorization Header

When api_base is set, the x-goog-api-key header is NOT sent even when api_key is provided.

Bug #2: Ignored Extra Headers

When api_base is set, the extra_headers parameter is completely ignored.

Test Output

================================================================================
LITELLM GEMINI EMBEDDING BUGS - DEMONSTRATION
================================================================================
LiteLLM Version: 1.80.10

================================================================================
BUG #1: Missing Authorization Header (x-goog-api-key)
================================================================================

EXPECTED BEHAVIOR:
  When api_key is provided, request should include:
    headers = {
      'Content-Type': 'application/json; charset=utf-8',
      'x-goog-api-key': 'AIzaSyAS0bSJ2TaEvFte...'  ← SHOULD BE SENT
    }

ACTUAL BEHAVIOR:
  Request only includes:
    headers = {
      'Content-Type': 'application/json; charset=utf-8'
      # ← x-goog-api-key is MISSING!
    }

✅ BUG CONFIRMED: Request failed with authentication error
   Error: litellm.BadRequestError: GeminiException BadRequestError - {
  "error": {
    "code": 403,
    "message": "Method doesn't allow unregistered callers (callers without
                established identity). Please use API Keys..."
  }
}

================================================================================
BUG #2: Ignored Extra Headers Parameter
================================================================================

EXPECTED BEHAVIOR:
  When extra_headers is provided, request should include:
    headers = {
      'Content-Type': 'application/json; charset=utf-8',
      'X-Custom-Header': 'test-value',  ← SHOULD BE SENT
      'Authorization': 'Bearer token'   ← SHOULD BE SENT
    }

ACTUAL BEHAVIOR:
  extra_headers parameter is completely ignored
  Request only includes Content-Type header

✅ BUG CONFIRMED: extra_headers were ignored
   If extra_headers worked, Authorization header would be sent
   Error: litellm.BadRequestError: GeminiException BadRequestError - {
  "error": {
    "code": 403,
    "message": "Method doesn't allow unregistered callers..."
  }
}

================================================================================
ROOT CAUSE
================================================================================
File: litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py
Method: GoogleBatchEmbeddings.batch_embeddings()

Lines 60-62 create headers WITHOUT including auth_header:
  headers = {'Content-Type': 'application/json; charset=utf-8'}
  # ← auth_header is retrieved but NEVER added!
  # ← extra_headers parameter is NEVER checked!

================================================================================
FIX (2 lines)
================================================================================
headers = {'Content-Type': 'application/json; charset=utf-8'}
if auth_header:
    headers.update(auth_header)
================================================================================

Summary

Both Bugs Confirmed

Bug #1: Missing Authorization Header

• Error Code: 403 Forbidden
• Root Cause: "Method doesn't allow unregistered callers (callers without established identity)"
• Impact: Cannot use Gemini embeddings with custom api_base (e.g., Cloudflare AI Gateway)

Bug #2: Ignored Extra Headers

• The extra_headers parameter is completely ignored
• No way to add custom headers when using custom api_base
• Prevents workarounds like manually adding Authorization header

Bug Location

File: litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py

Method: GoogleBatchEmbeddings.batch_embeddings()

Lines: 60-62 (creates headers without auth_header)

Proposed Fix

# Current (broken):
headers = {
    "Content-Type": "application/json; charset=utf-8",
}

# Fixed:
headers = {
    "Content-Type": "application/json; charset=utf-8",
}
if auth_header:
    headers.update(auth_header)

Workaround

Use the monkey patch from ai/tests/litellm_gemini_embedding_patch.py:

from litellm_gemini_embedding_patch import apply_litellm_gemini_embedding_patch

apply_litellm_gemini_embedding_patch()

# Now embedding() works with custom api_base

GitHub Issue

To report this bug to LiteLLM:

1. Go to: https://github.com/BerriAI/litellm/issues/new
2. Use the template from: ai/tests/LITELLM_GITHUB_ISSUE_QUICK_COPY.txt

Related Files

• ai/tests/confirm_bug.py - This test script
• ai/tests/litellm_gemini_embedding_patch.py - Workaround code
• ai/tests/LITELLM_GITHUB_ISSUE_QUICK_COPY.txt - GitHub issue template
• ai/PROXIES.md - Complete proxy architecture documentation

Test the bug

#!/usr/bin/env python3
"""
LiteLLM Bug Demonstration: Missing Headers with Custom api_base

Demonstrates TWO bugs in litellm.embedding() when using Gemini with custom api_base:
1. Missing Authorization Header: x-goog-api-key is NOT sent even when api_key is provided
2. Ignored Extra Headers: extra_headers parameter is completely ignored
"""
import litellm
import os
from dotenv import load_dotenv

load_dotenv()

# Get version
try:
    import importlib.metadata
    version = importlib.metadata.version('litellm')
except Exception:
    version = "unknown"

print("=" * 80)
print("LITELLM GEMINI EMBEDDING BUGS - DEMONSTRATION")
print("=" * 80)
print(f"LiteLLM Version: {version}")
print()

# Configuration
gemini_api_key = os.getenv("GEMINI_API_KEY")
account_id = os.getenv("PROXY_CFAIG_ACCOUNT_ID")
cfaig_url = f"https://gateway.ai.cloudflare.com/v1/{account_id}/noauth/google-ai-studio/v1beta"

print("=" * 80)
print("BUG #1: Missing Authorization Header (x-goog-api-key)")
print("=" * 80)
print()
print("EXPECTED BEHAVIOR:")
print("  When api_key is provided, request should include:")
print("    headers = {")
print("      'Content-Type': 'application/json; charset=utf-8',")
print(f"      'x-goog-api-key': '{gemini_api_key[:20]}...'  ← SHOULD BE SENT")
print("    }")
print()
print("ACTUAL BEHAVIOR:")
print("  Request only includes:")
print("    headers = {")
print("      'Content-Type': 'application/json; charset=utf-8'")
print("      # ← x-goog-api-key is MISSING!")
print("    }")
print()

try:
    response = litellm.embedding(
        model="gemini/gemini-embedding-001",
        input=["Test"],
        api_key=gemini_api_key,
        api_base=cfaig_url,
    )
    print("❌ UNEXPECTED: Request succeeded (bug may be fixed?)")
except Exception as e:
    error_msg = str(e)
    if "403" in error_msg or "unregistered callers" in error_msg or "established identity" in error_msg:
        print("✅ BUG CONFIRMED: Request failed with authentication error")
        print(f"   Error: {error_msg[:150]}...")
    else:
        print(f"⚠️  Unexpected error: {error_msg[:150]}...")

print()
print("=" * 80)
print("BUG #2: Ignored Extra Headers Parameter")
print("=" * 80)
print()
print("EXPECTED BEHAVIOR:")
print("  When extra_headers is provided, request should include:")
print("    headers = {")
print("      'Content-Type': 'application/json; charset=utf-8',")
print("      'X-Custom-Header': 'test-value',  ← SHOULD BE SENT")
print("      'Authorization': 'Bearer token'   ← SHOULD BE SENT")
print("    }")
print()
print("ACTUAL BEHAVIOR:")
print("  extra_headers parameter is completely ignored")
print("  Request only includes Content-Type header")
print()

try:
    response = litellm.embedding(
        model="gemini/gemini-embedding-001",
        input=["Test"],
        api_key=gemini_api_key,
        api_base=cfaig_url,
        extra_headers={
            "X-Custom-Header": "test-value",
            "Authorization": "Bearer fake-token"
        }
    )
    print("❌ UNEXPECTED: Request succeeded (bug may be fixed?)")
except Exception as e:
    error_msg = str(e)
    if "403" in error_msg or "unregistered callers" in error_msg:
        print("✅ BUG CONFIRMED: extra_headers were ignored")
        print("   If extra_headers worked, Authorization header would be sent")
        print(f"   Error: {error_msg[:150]}...")
    else:
        print(f"⚠️  Unexpected error: {error_msg[:150]}...")

print()
print("=" * 80)
print("ROOT CAUSE")
print("=" * 80)
print("File: litellm/llms/vertex_ai/gemini_embeddings/batch_embed_content_handler.py")
print("Method: GoogleBatchEmbeddings.batch_embeddings()")
print()
print("Lines 60-62 create headers WITHOUT including auth_header:")
print("  headers = {'Content-Type': 'application/json; charset=utf-8'}")
print("  # ← auth_header is retrieved but NEVER added!")
print("  # ← extra_headers parameter is NEVER checked!")
print()
print("=" * 80)
print("FIX (2 lines)")
print("=" * 80)
print("headers = {'Content-Type': 'application/json; charset=utf-8'}")
print("if auth_header:")
print("    headers.update(auth_header)")
print("=" * 80)