Migrate to Local Multi-Model Architecture (BLIP-2 + Whisper + CLIP + Phi-3.5)

[thewildofficial]

Problem

Currently, the project uses Google's Gemini 2.5 Flash API for Vision Language Model (VLM) analysis, which violates our constraint of not using external APIs. This dependency:

• Requires API keys and external network calls
• Creates privacy/security concerns
• Adds latency and costs
• Makes the system dependent on external services

Current Implementation:

• src/media/vlm_analyzer.py uses google-generativeai SDK
• requirements.txt includes google-generativeai==0.8.3
• Configuration requires GEMINI_API_KEY environment variable
• Model: gemini-2.5-flash or gemini-2.5-flash-lite

Revised Solution: Multi-Model Local Architecture

After evaluating hardware constraints (6GB VRAM available), we will use specialized lightweight models that collectively provide multimodal capabilities:

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    Backend Services Layer                    │
├─────────────────────────────────────────────────────────────┤
│  Image Analysis  │  Video Analysis  │  Audio Processing    │
│    Service       │     Service      │      Service         │
└────────┬─────────┴────────┬─────────┴──────────┬───────────┘
         │                  │                     │
    ┌────▼─────┐      ┌────▼─────┐         ┌────▼─────┐
    │  BLIP-2  │      │   CLIP   │         │ Whisper  │
    │ Caption  │      │ Embedder │         │  Base    │
    │ (~1.5GB) │      │ (~500MB) │         │ (~1.5GB) │
    └──────────┘      └──────────┘         └──────────┘
                            │
                    ┌───────▼────────┐
                    │ Phi-3.5-mini   │
                    │ 128K Context   │
                    │   (~2.5GB)     │
                    └────────────────┘
Total VRAM: ~5.5-6GB ✅

---

📦 Model Selection & Justification

1️⃣ Image Understanding: BLIP-2 (~1.5GB VRAM)

• Model: Salesforce/blip2-opt-2.7b or blip2-flan-t5-xl
• Purpose: Image captioning, visual question answering
• Capabilities:
  • Detailed natural language image descriptions
  • Scene understanding and object detection
  • Visual question answering
  • Context-aware captioning
• Performance: ~2-3s per image on GPU
• Output: Rich natural language descriptions

2️⃣ Image Embeddings: CLIP ViT-B-32 (~500MB VRAM)

• Purpose: Semantic embeddings for clustering & search
• Why Keep: Already implemented, lightweight, excellent quality
• Performance: ~100ms per image
• Output: 512-dim normalized vectors
• Note: No changes needed to existing clustering logic!

3️⃣ Audio Transcription: Whisper Base (~1.5GB VRAM)

• Model: openai/whisper-base or whisper-small
• Purpose: Speech-to-text, multilingual transcription
• Capabilities:
  • 99 language support
  • Automatic language detection
  • Timestamp generation
  • Robust to background noise
• Performance: ~1× real-time (1 min audio in ~60s)
• Output: Transcription + timestamps + language detection

4️⃣ Text Analysis & Reasoning: Phi-3.5-mini (~2.5GB VRAM) ⭐

• Model: microsoft/Phi-3.5-mini-instruct (3.8B parameters)
• Purpose:
  • Extract structured metadata from BLIP captions
  • Analyze Whisper transcriptions semantically
  • Directory structure reasoning & file organization
  • Semantic tag generation
  • Long document understanding

Why Phi-3.5-mini is Perfect:

• ✅ 128K context - Process entire documents/file collections in one pass
• ✅
• ✅ Quantization aware - FP8 inference without performance loss
• ✅ Resource efficient - Runs on consumer laptops with 6GB VRAM
• ✅ Better metadata extraction - Understands context vs. purely statistical approaches
• ✅ Outperforms larger models - Beats Llama 3.1 8B on complex reasoning despite fewer parameters

Performance: ~50-100 tokens/s on 6GB GPU

---

🔄 Implementation Pipeline

Image Analysis Workflow

Image → BLIP-2 Caption → Phi-3.5 Metadata Extraction → Structured JSON
                    ↓
              CLIP Embedding →

Video Analysis Workflow

Video → Keyframes → BLIP-2 Captions (per frame)
                         ↓
              Phi-3.5 Temporal Analysis → Scene Understanding
                         ↓
              CLIP Embeddings → Video-level Embedding

Audio Analysis Workflow

Audio → Whisper Transcription → Phi-3.5 Semantic Analysis
                                      ↓
                              Tags + Summary + Classification

File Organization Workflow

Files → Extract Metadata (BLIP/Whisper) → Phi-3.5 128K Context
                                                ↓
                                   Analyze entire collection
                                                ↓
                                   Suggest directory structure

---

📊 VRAM Budget Breakdown

Model	VRAM (FP16)	VRAM (FP8/INT8)	Purpose
CLIP ViT-B-32	~500MB	~300MB	Embeddings
BLIP-2 OPT-2.7B	~1.5GB	~1GB	Image captions
Whisper Base	~1.5GB	~1GB	Audio transcription
Phi-3.5-mini (FP8)	~2.5GB	~2GB	Text reasoning
Total (FP16)	~6GB	~4.5GB	✅ Within budget

Optimization: Can use quantized versions to free up memory for larger batches or concurrent requests.

---

🐳 Docker Deployment Architecture

version: '3.8'

services:
  app:
    build: .
    ports:
      - "8080:8080"
    depends_on:
      - blip-server
      - whisper-server
      - phi35-server
    environment:
      - BLIP_ENDPOINT=http://blip-server:8001
      - WHISPER_ENDPOINT=http://whisper-server:8002
      - PHI35_ENDPOINT=http://phi35-server:8003
      - DATABASE_URL=postgresql://user:pass@postgres:5432/mammothbox

  # BLIP-2 Image Captioning Service
  blip-server:
    build: ./services/blip
    ports:
      - "8001:8001"
    volumes:
      - model_cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
        limits:
          memory: 4G

  # Whisper Audio Transcription Service
  whisper-server:
    build: ./services/whisper
    ports:
      - "8002:8002"
    volumes:
      - model_cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
        limits:
          memory: 4G

  # Phi-3.5-mini Text Reasoning Service
  phi35-server:
    image: vllm/vllm-openai:latest
    ports:
      - "8003:8003"
    environment:
      - MODEL_NAME=microsoft/Phi-3.5-mini-instruct
    volumes:
      - model_cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    command:
      - --model
      - microsoft/Phi-3.5-mini-instruct
      - --max-model-len
      - "131072"
      - --port
      - "8003"
      - --dtype
      - auto
      - --gpu-memory-utilization
      - "0.4"

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: mammothbox
      POSTGRES_USER: user
      POSTGRES_PASSWORD: pass
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  postgres_data:
  model_cache:

---

📋 Migration Roadmap

See individual phase issues for detailed tasks:

• Phase 1: Setup Multi-Model Infrastructure (BLIP-2 + Whisper + Phi-3.5) #43 - Phase 1: Infrastructure Setup
• Phase 2: Migrate Image Analysis to BLIP-2 + Phi-3.5 #44 - Phase 2: Image Analysis Migration
• Phase 3: Implement Audio Processing (Whisper + Phi-3.5) #45 - Phase 3: Audio Processing Implementation
• Phase 4: Intelligent File Organization with Phi-3.5 (128K Context) #46 - Phase 4: Semantic Grouping & File Organization

---

✅ Success Criteria

• ✅ No external API calls required
• ✅ All models run locally on 6GB VRAM
• ✅ Image analysis accuracy >90% of Gemini baseline
• ✅ Audio transcription accuracy >95% for clear speech
• ✅ Inference time <5s per media file
• ✅ Total memory footprint ≤6GB VRAM
• ✅ File organization suggestions >80% user acceptance
• ✅ All existing tests passing
• ✅ 128K context enables whole-collection analysis

---

🎯 Key Advantages

1. Hardware Optimized: Precisely fits 6GB VRAM budget
2. Modular Architecture: Independent microservices, easy to debug/scale
3. Best-in-Class Models: Each model is optimized for its specific task
4. Future-Proof: Can upgrade individual models without system rewrite
5. No External Dependencies: Fully local, private, no API costs
6. Production Ready: All models battle-tested and widely adopted
7. Exceptional Context: Phi-3.5's 128K tokens enables sophisticated reasoning
8. Keep What Works: CLIP embeddings/clustering unchanged!

---

📚 References

• BLIP-2: https://github.com/salesforce/LAVIS/tree/main/projects/blip2
• CLIP: https://github.com/openai/CLIP
• Whisper: https://github.com/openai/whisper
• Phi-3.5: https://huggingface.co/microsoft/Phi-3.5-mini-instruct
• vLLM: https://docs.vllm.ai/
• Sentence Transformers: https://www.sbert.net/

[thewildofficial]

Feasible Integration Architecture Using Gemma 3n E4B-IT

Based on the comprehensive capabilities of Gemma 3n E4B-IT, here's how we can feasibly integrate this unified multimodal model across our entire backend system:

---

🏗️ Unified Architecture Overview

Instead of maintaining separate specialized models (CLIP for images, Whisper for audio, separate embedders), we can consolidate to a single Gemma 3n E4B-IT endpoint that handles all multimodal tasks. This drastically simplifies our architecture:

┌─────────────────────────────────────────────────────────────┐
│                    Backend Services Layer                    │
├─────────────────────────────────────────────────────────────┤
│  Image Analysis  │  Video Analysis  │  Audio Processing    │
│    Service       │     Service      │      Service         │
└────────┬─────────┴────────┬─────────┴──────────┬───────────┘
         │                  │                     │
         └──────────────────┼─────────────────────┘
                            │
                   ┌────────▼─────────┐
                   │   Gemma 3n E4B   │
                   │  Unified Model   │
                   │   (Docker/vLLM)  │
                   └──────────────────┘

---

📋 Service-by-Service Integration Plan

1️⃣ Image Analysis Service (src/media/vlm_analyzer.py)

Current: Uses Gemini API for VLM analysis
Migration: Replace with Gemma 3n's MobileNet v5 vision encoder

Implementation:

# src/media/vlm_analyzer.py (refactored)
import requests
from PIL import Image

class VLMAnalyzer:
    def __init__(self):
        self.gemma_endpoint = "http://gemma3n-server:8000/generate"
    
    async def analyze_image(self, image_path: str) -> VLMMetadata:
        """Analyze image using local Gemma 3n model."""
        with open(image_path, 'rb') as f:
            files = {'image': f}
            data = {
                'prompt': self.IMAGE_ANALYSIS_PROMPT,
                'max_tokens': 500
            }
            response = requests.post(self.gemma_endpoint, files=files, data=data)
            
        # Parse structured JSON response
        result = response.json()['response']
        return VLMMetadata(**json.loads(result))
    
    async def label_cluster(self, image_paths: List[str]) -> ClusterMetadata:
        """Multi-image cluster labeling with Gemma 3n."""
        # Gemma 3n supports multiple images in single request
        files = [('image', open(path, 'rb')) for path in image_paths[:5]]
        data = {
            'prompt': self.CLUSTER_LABELING_PROMPT_TEMPLATE,
            'max_tokens': 300
        }
        response = requests.post(self.gemma_endpoint, files=files, data=data)
        return ClusterMetadata(**json.loads(response.json()['response']))

Benefits:

• ✅ No API keys required
• ✅ Same structured JSON output as Gemini
• ✅ 256-768px resolution support (matches current implementation)
• ✅ Multi-image analysis (cluster labeling already supported)

---

2️⃣ Video Analysis Service (src/media/processor.py)

Current: Extracts keyframes, analyzes each with VLM
Migration: Use Gemma 3n's native video understanding

Implementation:

# src/media/processor.py
class VideoProcessor:
    async def analyze_video(self, video_path: str) -> VideoMetadata:
        """Analyze video using Gemma 3n frame understanding."""
        # Extract keyframes (existing logic)
        keyframes = self._extract_keyframes(video_path, num_frames=8)
        
        # Send all frames to Gemma 3n for temporal analysis
        files = [('image', open(frame, 'rb')) for frame in keyframes]
        data = {
            'prompt': '''Analyze this video sequence. Provide:
            1. Scene description and temporal flow
            2. Primary activities/actions
            3. Key objects and their movements
            4. Suggested categorization
            Return JSON format.''',
            'max_tokens': 600
        }
        
        response = requests.post(
            "http://gemma3n-server:8000/generate",
            files=files,
            data=data
        )
        
        return self._parse_video_metadata(response.json()['response'])

Benefits:

• ✅ Temporal understanding across frames
• ✅ Action recognition from video sequences
• ✅ Reduced processing time (batch frame analysis)

---

3️⃣ Audio Understanding & Transcription (NEW: src/audio/processor.py)

Current: Not implemented
Migration: Use Gemma 3n's USM-based audio encoder

Implementation:

# src/audio/processor.py (new module)
from dataclasses import dataclass
from typing import List

@dataclass
class AudioMetadata:
    transcription: str
    language: str
    semantic_tags: List[str]
    content_summary: str
    speaker_count: int
    audio_type: str  # speech, music, ambient, mixed

class AudioProcessor:
    def __init__(self):
        self.gemma_endpoint = "http://gemma3n-server:8000/generate"
    
    async def transcribe_and_analyze(self, audio_path: str) -> AudioMetadata:
        """Transcribe and extract semantic meaning from audio."""
        with open(audio_path, 'rb') as f:
            files = {'audio': f}
            data = {
                'prompt': '''Transcribe this audio and provide:
                1. Full transcription
                2. Detected language
                3. 5-10 semantic tags (topics, entities, keywords)
                4. Brief content summary
                5. Estimated speaker count
                6. Audio type classification
                Return JSON format.''',
                'max_tokens': 1000
            }
            response = requests.post(self.gemma_endpoint, files=files, data=data)
        
        return AudioMetadata(**json.loads(response.json()['response']))
    
    async def generate_audio_embeddings(self, audio_path: str) -> np.ndarray:
        """Generate semantic embeddings for audio content."""
        # Gemma 3n USM encoder: 6.25 tokens/second
        # Use embedding extraction endpoint
        response = requests.post(
            "http://gemma3n-server:8000/embed",
            files={'audio': open(audio_path, 'rb')}
        )
        return np.array(response.json()['embedding'])

Benefits:

• ✅ 140+ language support (vs Whisper's ~100)
• ✅ Unified embeddings with visual modalities
• ✅ Semantic understanding beyond transcription

---

4️⃣ Semantic Grouping & File Structuring (ENHANCED: src/media/clusterer.py)

Current: Uses CLIP embeddings for clustering
Migration: Use Gemma 3n's unified multimodal embeddings

Implementation:

# src/media/clusterer.py (enhanced)
class MediaClusterer:
    async def generate_unified_embeddings(
        self, 
        files: List[MediaFile]
    ) -> Dict[str, np.ndarray]:
        """Generate embeddings for images, videos, audio using single model."""
        embeddings = {}
        
        for file in files:
            if file.type == 'image':
                embedding = await self._embed_image(file.path)
            elif file.type == 'video':
                # Embed keyframes and average
                frames = self._extract_keyframes(file.path)
                frame_embeddings = [
                    await self._embed_image(f) for f in frames
                ]
                embedding = np.mean(frame_embeddings, axis=0)
            elif file.type == 'audio':
                embedding = await self._embed_audio(file.path)
            
            embeddings[file.id] = embedding
        
        return embeddings
    
    async def _embed_image(self, image_path: str) -> np.ndarray:
        response = requests.post(
            "http://gemma3n-server:8000/embed",
            files={'image': open(image_path, 'rb')}
        )
        return np.array(response.json()['embedding'])
    
    async def suggest_directory_structure(
        self, 
        embeddings: Dict[str, np.ndarray],
        metadata: Dict[str, dict]
    ) -> Dict[str, str]:
        """Use Gemma 3n's reasoning to suggest optimal directory structure."""
        
        # Prepare metadata summary for LLM reasoning
        summary = self._prepare_metadata_summary(metadata)
        
        prompt = f'''Given these {len(metadata)} files with the following metadata:
{summary}

Suggest an optimal directory structure that:
1. Groups semantically similar files
2. Uses clear, descriptive folder names
3. Supports easy retrieval and organization
4. Considers content type, topics, and temporal aspects

Return JSON with mapping: {{"file_id": "suggested/path/"}}'''
        
        response = requests.post(
            "http://gemma3n-server:8000/generate",
            json={'prompt': prompt, 'max_tokens': 2000}
        )
        
        return json.loads(response.json()['response'])

Benefits:

• ✅ Single embedding space for all modalities
• ✅ Better cross-modal similarity (e.g., image + audio of same scene)
• ✅ LLM reasoning for intelligent file organization
• ✅ 32K context window for processing large file collections

---

🐳 Docker Deployment Architecture

docker-compose.yml:

version: '3.8'

services:
  # Main application
  app:
    build: .
    ports:
      - "8080:8080"
    depends_on:
      - gemma3n-server
      - postgres
      - redis
    environment:
      - GEMMA_ENDPOINT=http://gemma3n-server:8000
      - DATABASE_URL=postgresql://user:pass@postgres:5432/mammothbox
    volumes:
      - ./storage:/app/storage

  # Unified Gemma 3n model server
  gemma3n-server:
    image: vllm/vllm-openai:latest
    ports:
      - "8000:8000"
    environment:
      - MODEL_NAME=google/gemma-3n-e4b-it
      - HF_TOKEN=${HF_TOKEN}
      - CUDA_VISIBLE_DEVICES=0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - ./model_cache:/root/.cache/huggingface
    command:
      - --model
      - google/gemma-3n-e4b-it
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --dtype
      - bfloat16
      - --max-model-len
      - "32768"
      - --gpu-memory-utilization
      - "0.9"

  postgres:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=mammothbox
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

volumes:
  postgres_data:
  model_cache:

---

📊 Performance Estimates

Based on benchmarks with 16GB VRAM GPU:

Task	Current (Gemini API)	Gemma 3n E4B Local	Improvement
Single image analysis	~2s (network + API)	~0.5s	4× faster
Video keyframe batch (8 frames)	~16s (sequential)	~2s (batched)	8× faster
Audio transcription (1 min)	N/A	~10s	New capability
Cluster labeling (5 images)	~10s	~1.5s	6.6× faster
Embedding generation	Separate CLIP call	Unified call	1 endpoint

Memory footprint: ~8-12GB VRAM (fits on single consumer GPU)
Throughput: ~20-30 requests/second with vLLM optimizations

---

🔄 Migration Strategy

Phase 1: Infrastructure (Week 1)

• Set up Docker deployment with vLLM
• Create unified API wrapper for Gemma 3n
• Implement embedding extraction endpoints
• Add health checks and monitoring

Phase 2: Image/Video Migration (Week 2)

• Refactor vlm_analyzer.py to use local Gemma endpoint
• Update processor.py for video analysis
• Migrate embedding generation in embedder.py
• Run A/B tests against Gemini baseline

Phase 3: Audio Integration (Week 3)

• Implement new audio/processor.py module
• Add audio transcription endpoints
• Create audio embedding pipeline
• Integrate with media service

Phase 4: Semantic Grouping (Week 4)

• Enhance clusterer.py with unified embeddings
• Implement directory structure suggestion logic
• Add user feedback loop for file organization
• Create configuration for organization rules

Phase 5: Testing & Optimization (Week 5)

• End-to-end integration tests
• Performance benchmarking
• Memory optimization (quantization if needed)
• Documentation updates

---

✅ Why This Integration Makes Sense

1. Single Model, All Modalities: Eliminates complexity of managing CLIP + BLIP + Whisper + separate embedders
2. Unified Embedding Space: Better cross-modal understanding (e.g., find images related to audio description)
3. Cost Elimination: No Gemini API costs, no rate limits
4. Privacy & Security: All processing on-premise, no data leaves your infrastructure
5. Reduced Latency: Local inference is 4-8× faster than API calls
6. Scalability: Can deploy multiple Gemma instances behind load balancer for high throughput
7. 32K Context Window: Process large batches of files in single request for intelligent grouping

---

🎯 Next Steps

1. Create separate issues for each service migration (see below)
2. Set up dev environment with Docker deployment
3. Start with image analysis migration (smallest scope)
4. Iteratively add audio and enhanced semantic grouping
5. Continuously benchmark against current Gemini baseline

This architecture gives us a production-ready, scalable, unified multimodal backend that eliminates external dependencies while improving performance. 🚀

[thewildofficial]

🔄 Updated Migration Strategy: Multi-Model Approach for 6GB VRAM

After evaluating resource constraints (6GB VRAM available), Gemma 3n E4B-IT (requires 8-12GB) is not feasible for our hardware.

✅ Revised Approach: Specialized Local Models

Instead of a unified model, we'll use specialized lightweight models that collectively provide multimodal capabilities while fitting within our 6GB VRAM budget:

---

🏗️ Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                    Backend Services Layer                    │
├─────────────────────────────────────────────────────────────┤
│  Image Analysis  │  Video Analysis  │  Audio Processing    │
│    Service       │     Service      │      Service         │
└────────┬─────────┴────────┬─────────┴──────────┬───────────┘
         │                  │                     │
    ┌────▼─────┐      ┌────▼─────┐         ┌────▼─────┐
    │   BLIP-2 │      │   CLIP   │         │ Whisper  │
    │ Caption  │      │ Embedder │         │  Base    │
    │ (~1.5GB) │      │ (~500MB) │         │ (~1.5GB) │
    └──────────┘      └──────────┘         └──────────┘
                            │
                       ┌────▼─────┐
                       │ Local LLM│
                       │  Phi-3   │
                       │ (~2.5GB) │
                       └──────────┘

---

📦 Model Selection & Resource Allocation

1️⃣ Image Understanding: BLIP-2 (~1.5GB VRAM)

• Model: Salesforce/blip2-opt-2.7b or blip2-flan-t5-xl
• Purpose: Image captioning, visual question answering
• Capabilities:
  • Detailed image descriptions
  • Scene understanding
  • Object detection descriptions
  • Visual question answering
• Performance: ~2-3s per image
• Output: Natural language descriptions + structured metadata

2️⃣ Image Embeddings: CLIP (~500MB VRAM)

• Purpose: Semantic embeddings for clustering & search
• Keep existing implementation - it's lightweight and works well
• Performance: ~100ms per image
• Output: 512-dim normalized vectors

3️⃣ Audio Transcription: Whisper Base (~1.5GB VRAM)

• Model: openai/whisper-base or whisper-small
• Purpose: Speech-to-text, multilingual transcription
• Capabilities:
  • 99 language support
  • Automatic language detection
  • Timestamp generation
  • Decent accuracy for speech
• Performance: ~1× real-time (1 min audio in ~60s on CPU, faster on GPU)
• Output: Transcription + timestamps + language

4️⃣ Text Analysis & Reasoning: Phi-3 Mini (~2.5GB VRAM)

• Model: microsoft/Phi-3-mini-4k-instruct (3.8B parameters)
• Purpose: Text understanding, semantic analysis, reasoning
• Capabilities:
  • Analyze BLIP captions and extract structured metadata
  • Suggest directory structures based on content
  • Generate tags from descriptions
  • Summarize audio transcriptions
  • Semantic grouping logic
• Performance: ~50-100 tokens/s on 6GB GPU
• Context: 4K tokens
• Output: Structured JSON, summaries, classifications

---

🔄 Service-by-Service Implementation

Image Analysis Pipeline

# src/media/vlm_analyzer.py (refactored)

class VLMAnalyzer:
    def __init__(self):
        self.blip_endpoint = "http://blip-server:8001"
        self.clip_embedder = MediaEmbedder()  # Keep existing CLIP
        self.llm_endpoint = "http://phi3-server:8003"
    
    async def analyze_individual_image(self, image_path: str) -> VLMMetadata:
        """Multi-stage image analysis."""
        
        # Stage 1: Get BLIP caption
        caption = await self._get_blip_caption(image_path)
        
        # Stage 2: Use Phi-3 to extract structured metadata from caption
        structured_data = await self._extract_metadata_from_caption(caption)
        
        # Stage 3: Generate CLIP embedding for clustering
        embedding = self.clip_embedder.encode_image(Image.open(image_path))
        
        return VLMMetadata(
            primary_category=structured_data['category'],
            tags=structured_data['tags'],
            description=caption,
            detected_objects=structured_data['objects'],
            scene_type=structured_data['scene_type'],
            color_palette=self._extract_colors(image_path),
            suggested_cluster_name=structured_data['cluster_name'],
            vlm_model="blip2+phi3",
            fallback_used=False
        )
    
    async def _get_blip_caption(self, image_path: str) -> str:
        """Get detailed caption from BLIP-2."""
        with open(image_path, 'rb') as f:
            response = requests.post(
                f"{self.blip_endpoint}/caption",
                files={'image': f}
            )
        return response.json()['caption']
    
    async def _extract_metadata_from_caption(self, caption: str) -> dict:
        """Use Phi-3 to extract structured metadata from BLIP caption."""
        prompt = f"""Analyze this image caption and extract structured metadata:
Caption: "{caption}"

Return JSON with:
{{
    "category": "main subject category",
    "tags": ["5-10 descriptive tags"],
    "objects": [{{"name": "object", "confidence": 0.9}}],
    "scene_type": "portrait|landscape|indoor|outdoor|etc",
    "cluster_name": "2-4 word grouping name"
}}"""
        
        response = requests.post(
            f"{self.llm_endpoint}/generate",
            json={'prompt': prompt, 'max_tokens': 300}
        )
        return json.loads(response.json()['text'])

Video Analysis Pipeline

# src/media/processor.py

class VideoProcessor:
    async def analyze_video(self, video_path: str) -> VideoMetadata:
        """Analyze video using keyframes."""
        
        # Extract keyframes
        keyframes = self._extract_keyframes(video_path, num_frames=8)
        
        # Get BLIP captions for each keyframe
        captions = []
        for frame in keyframes:
            caption = await self._get_blip_caption(frame)
            captions.append(caption)
        
        # Use Phi-3 to analyze temporal flow and summarize
        summary_prompt = f"""Analyze this video sequence from {len(captions)} keyframes:

{chr(10).join([f'Frame {i+1}: {cap}' for i, cap in enumerate(captions)])}

Provide:
1. Overall video description
2. Main activities/actions
3. Scene progression
4. Suggested categorization

Return JSON format."""
        
        response = requests.post(
            f"{self.llm_endpoint}/generate",
            json={'prompt': summary_prompt, 'max_tokens': 500}
        )
        
        return self._parse_video_metadata(response.json()['text'])

Audio Processing Pipeline

# src/audio/processor.py (new)

class AudioProcessor:
    def __init__(self):
        self.whisper_endpoint = "http://whisper-server:8002"
        self.llm_endpoint = "http://phi3-server:8003"
    
    async def transcribe_and_analyze(self, audio_path: str) -> AudioMetadata:
        """Transcribe and extract semantic meaning."""
        
        # Stage 1: Whisper transcription
        transcription = await self._transcribe_audio(audio_path)
        
        # Stage 2: Phi-3 semantic analysis
        analysis = await self._analyze_transcription(transcription['text'])
        
        return AudioMetadata(
            transcription=transcription['text'],
            language=transcription['language'],
            semantic_tags=analysis['tags'],
            content_summary=analysis['summary'],
            speaker_count=analysis['speaker_count'],
            audio_type=analysis['audio_type']
        )
    
    async def _transcribe_audio(self, audio_path: str) -> dict:
        """Transcribe with Whisper."""
        with open(audio_path, 'rb') as f:
            response = requests.post(
                f"{self.whisper_endpoint}/transcribe",
                files={'audio': f}
            )
        return response.json()
    
    async def _analyze_transcription(self, text: str) -> dict:
        """Use Phi-3 to analyze transcription content."""
        prompt = f"""Analyze this audio transcription:

"{text}"

Extract:
1. 5-10 semantic tags (topics, keywords, entities)
2. Brief summary (1-2 sentences)
3. Estimated speaker count
4. Audio type (speech/conversation/lecture/podcast/etc)

Return JSON."""
        
        response = requests.post(
            f"{self.llm_endpoint}/generate",
            json={'prompt': prompt, 'max_tokens': 300}
        )
        return json.loads(response.json()['text'])

---

🐳 Docker Deployment

version: '3.8'

services:
  app:
    build: .
    ports:
      - "8080:8080"
    depends_on:
      - blip-server
      - whisper-server
      - phi3-server
    environment:
      - BLIP_ENDPOINT=http://blip-server:8001
      - WHISPER_ENDPOINT=http://whisper-server:8002
      - PHI3_ENDPOINT=http://phi3-server:8003

  # BLIP-2 for image captioning (~1.5GB VRAM)
  blip-server:
    image: pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime
    ports:
      - "8001:8001"
    volumes:
      - ./model_cache:/root/.cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    command: python3 /app/blip_server.py

  # Whisper for audio transcription (~1.5GB VRAM)
  whisper-server:
    image: pytorch/pytorch:2.1.0-cuda11.8-cudnn8-runtime
    ports:
      - "8002:8002"
    volumes:
      - ./model_cache:/root/.cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    command: python3 /app/whisper_server.py

  # Phi-3 for text reasoning (~2.5GB VRAM)
  phi3-server:
    image: vllm/vllm-openai:latest
    ports:
      - "8003:8003"
    environment:
      - MODEL_NAME=microsoft/Phi-3-mini-4k-instruct
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
    command:
      - --model
      - microsoft/Phi-3-mini-4k-instruct
      - --port
      - "8003"
      - --gpu-memory-utilization
      - "0.4"

  postgres:
    image: postgres:15-alpine
    environment:
      POSTGRES_DB: mammothbox
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:
  model_cache:

---

📊 VRAM Budget Breakdown

Model	VRAM Usage	Purpose
BLIP-2 OPT-2.7B	~1.5GB	Image captioning
Whisper Base	~1.5GB	Audio transcription
Phi-3 Mini 4K	~2.5GB	Text reasoning & structuring
Total	~6GB	✅ Fits budget

Note: Models can share GPU with sequential loading or use CPU offloading for non-critical paths.

---

✅ Advantages of This Approach

1. Fits Hardware: Total ~6GB VRAM (within constraints)
2. Proven Models: All battle-tested, production-ready
3. Modular: Can upgrade individual models independently
4. Scalable: Can distribute across multiple machines later
5. No External APIs: Fully local and private
6. Best-in-class: Each model is optimized for its specific task

---

📋 Updated Migration Tasks

Phase 1: Infrastructure

• Set up BLIP-2 server with FastAPI
• Set up Whisper server with FastAPI
• Set up Phi-3 server with vLLM
• Docker compose with resource limits

Phase 2: Image Analysis

• Refactor vlm_analyzer.py to use BLIP+Phi-3 pipeline
• Keep CLIP for embeddings/clustering
• Remove Gemini API calls
• Update tests

Phase 3: Audio Processing

• Implement Whisper transcription service
• Integrate Phi-3 for semantic analysis
• Create audio embeddings (can use CLIP on spectrograms or dedicated audio model)

Phase 4: Semantic Grouping

• Use Phi-3 for directory structure suggestions
• Implement file organization logic
• Keep existing CLIP-based clustering

---

🎯 Why This Works Better

• Resource Efficient: Each model loads only when needed
• Higher Quality: BLIP-2 captions + Phi-3 reasoning > single model for our use case
• Flexibility: Can run models on CPU if GPU is busy
• Proven Stack: No experimental models, all widely adopted
• Easy Debugging: Isolated services easier to monitor/fix

This approach gives us multimodality within hardware constraints while maintaining high quality! 🚀