Cognesys — a self-improving RAG platform

A production-grade, enterprise-ready Retrieval-Augmented Generation (RAG) system with built-in self-improvement capabilities through AI agents, A/B testing, and continuous optimization.

Overview

This RAG system goes beyond simple retrieval and generation by implementing a complete self-improvement loop:

┌─────────────────────────────────────────────────────────────────┐
│                    SELF-IMPROVING RAG SYSTEM                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐  │
│  │ Document │───▶│ Chunking │───▶│Embedding │───▶│  Vector  │  │
│  │ Ingestion│    │ Pipeline │    │Generation│    │  Store   │  │
│  └──────────┘    └──────────┘    └──────────┘    └──────────┘  │
│                                                        │         │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐         │         │
│  │  Query   │◀───│  Hybrid  │◀───│ Reranker │◀────────┘         │
│  │ Rewriter │    │ Retrieval│    │          │                   │
│  └──────────┘    └──────────┘    └──────────┘                   │
│        │                                                         │
│        ▼                                                         │
│  ┌──────────┐    ┌──────────┐    ┌──────────┐                   │
│  │ Context  │───▶│   LLM    │───▶│ Response │                   │
│  │ Builder  │    │Generator │    │          │                   │
│  └──────────┘    └──────────┘    └──────────┘                   │
│                        │                                         │
│                        ▼                                         │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              SELF-IMPROVEMENT LOOP                       │    │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌─────────┐ │    │
│  │  │Evaluation│─▶│ Feedback │─▶│Optimizer │─▶│A/B Test │ │    │
│  │  │  Agents  │  │Collection│  │  Agents  │  │Framework│ │    │
│  │  └──────────┘  └──────────┘  └──────────┘  └─────────┘ │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                  │
└─────────────────────────────────────────────────────────────────┘

Key Features

Feature	Description
Multi-LLM Support	OpenAI, Azure OpenAI, Anthropic Claude, Ollama (local)
Hybrid Retrieval	Vector search + BM25 with configurable fusion
Smart Chunking	Fixed, recursive, semantic, and code-aware strategies
Auto-Evaluation	AI agents assess every response for quality
A/B Testing	Built-in experiment framework with a live activity timeline per experiment
Knowledge Graphs	Build named graphs from documents — LLM entity/relationship extraction, pgvector entity resolution, interactive React Flow visualization
Self-Improvement	Continuous optimization through AI analysis
Web UI	Next.js 16 app: public landing page, dashboard, search, documents, knowledge graph, evaluations, experiments, settings
Per-User Auth	JWT (httpOnly cookies), per-user Fernet-encrypted provider keys & model selection, password reset
Operational Metrics	JSON metrics endpoints + structured logging with per-request correlation IDs

Architecture

The diagram below covers every layer of the system — click any node in the interactive /architecture page of the UI for a full drill-down on any component.

flowchart TB
    %% ─────────────────────────────────────────────────────────────────────
    %% AUTH & ACCESS CONTROL
    %% ─────────────────────────────────────────────────────────────────────
    subgraph AUTH["🔐  Auth & Access Control"]
        direction LR
        Browser["Next.js UI\n/login · /register · /settings"]
        AuthSvc["Registration & Login\nbcrypt · Redis rate-limit\nhttpOnly cookie issuance"]
        JWTMgr["JWT Manager\nHS256 access token · 30 min\nrefresh rotation · Redis jti revocation"]
        UserCfg["User Settings\nFernet-encrypted provider keys\nllm_provider · embedding_provider\nmodel selection per user"]
        Browser -- "email + password" --> AuthSvc
        AuthSvc -- "issues access + refresh pair" --> JWTMgr
        JWTMgr -- "Set-Cookie: access_token\nSet-Cookie: refresh_token" --> Browser
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% DOCUMENT INGESTION
    %% ─────────────────────────────────────────────────────────────────────
    subgraph INGEST["📥  Document Ingestion  (async · BackgroundTask)"]
        direction LR
        Upload["Upload Endpoint\nPOST /documents\nPDF · DOCX · MD · HTML · CSV · Code\nSHA-256 dedup check"]
        FileProc["File Processor\nformat-specific extraction\ncharset detection · chardet"]
        Chunker["Smart Chunker\nFixed · Recursive · Semantic\nCode-aware · plugin registry"]
        Embedder["Embedding Manager\nOpenAI · Cohere · SentenceTransformers\nbatch 100 · Redis cache\nper-user key injection"]
        Upload --> FileProc --> Chunker --> Embedder
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% STORAGE LAYER
    %% ─────────────────────────────────────────────────────────────────────
    subgraph STORE["🗄️  Storage Layer  (polyglot persistence)"]
        direction LR
        PG[("PostgreSQL\ndocuments · chunks · evaluations\nexperiments · prompts · system_config\nusers · user_settings · provider_keys")]
        QD[("Qdrant\nvector embeddings\ncosine search · payload filter\nuser_id scoped")]
        RD[("Redis\nBM25 inverted index\nresponse cache · JWT jti store\nlogin rate-limit counters")]
        MG[("MongoDB\nevent logs · agent traces\noperational audit trail")]
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% QUERY PIPELINE
    %% ─────────────────────────────────────────────────────────────────────
    subgraph QUERY["🔍  Query Pipeline"]
        direction LR
        Gate["JWT Auth Gate\ncookie → Bearer → X-Api-Key\nresolves User object"]
        Rewriter["Query Rewriter\nAI agent · 2–3 semantic variants\nhandles typos + ambiguity"]
        Hybrid["Hybrid Retriever\nVectorRetriever Qdrant\nBM25Retriever Redis\nRRF or linear fusion · HYBRID_ALPHA"]
        Reranker["Cross-Encoder Reranker\nms-marco-MiniLM-L-6-v2\ntop-RERANK_TOP_K candidates"]
        CtxBuilder["Context Builder\ncontent dedup · tiktoken budget\nsource attribution + numbering"]
        LLMFact["LLM Factory\nOpenAI · Anthropic · Azure · Ollama\nper-user key · cache by key fingerprint\nactive prompt version from DB"]
        Response["RAG Response\nanswer · sources · query_id\nexperiment_id · latency_ms"]
        Gate --> Rewriter --> Hybrid --> Reranker --> CtxBuilder --> LLMFact --> Response
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% EVALUATION ENGINE
    %% ─────────────────────────────────────────────────────────────────────
    subgraph EVAL["📊  Evaluation Engine  (5 dims · asyncio.gather)"]
        direction LR
        EvalOrch["Orchestrator\nfires after every response\nstructured Pydantic output"]
        subgraph DIMS["Evaluators"]
            direction TB
            Rel["🎯 Relevance\nper-chunk scoring\nthreshold 0.70"]
            Hal["🛡️ Hallucination\nclaim-level · inverted\nmax 0.35"]
            Cit["📎 Citation\nsource attribution\naccuracy check"]
            Qual["⭐ Quality\nclarity · completeness\ncoherence · conciseness"]
            Ctx["📚 Context Quality\ncoverage · redundancy\nordering · sufficiency"]
        end
        EvalOrch --> DIMS
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% SELF-IMPROVEMENT LOOP
    %% ─────────────────────────────────────────────────────────────────────
    subgraph LOOP["⚡  Self-Improvement Loop  (fully automated)"]
        direction LR
        Feedback["Feedback Collection\nthumbs · 1–5 rating\ncorrection text · query_id link"]
        AutoOpt["Auto-Optimizer\nruns every N hours\nreads 7-day eval averages\nCohen's h ≥ 0.1 to promote"]
        ABTest["A/B Experiment Manager\nhash query_id % 100 routing\nDRAFT → RUNNING → COMPLETED\nvariant_config overrides"]
        SysCfg["System Config Store\nlive parameter overrides\ntop_k · hybrid_alpha · system_prompt\n5-min in-memory cache"]
        Feedback --> AutoOpt --> ABTest --> SysCfg
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% KNOWLEDGE GRAPH
    %% ─────────────────────────────────────────────────────────────────────
    subgraph KG["🕸️  Knowledge Graph  (on-demand · BackgroundTask)"]
        direction LR
        KGBuild["Graph Builder\nKnowledgeGraphService\nstatus empty→building→ready"]
        KGExtract["Extraction Agent\nLLM entities + relationships\nper chunk · JSON"]
        KGResolve["Entity Resolution\nnormalize + pgvector cosine merge\nKG_ENTITY_SIM_THRESHOLD"]
        KGStore[("KG Store\nkg_entities pgvector · kg_relationships\nkg_entity_mentions")]
        KGViz["Graph Explorer UI\n/knowledge-graph · React Flow\nclick node → detail panel"]
        KGBuild --> KGExtract --> KGResolve --> KGStore --> KGViz
    end

    %% Knowledge-graph connections
    PG -- "document chunks" --> KGBuild
    KGExtract -- "LLM extraction calls" --> LLMFact
    KGResolve -- "entity embeddings" --> Embedder

    %% ─────────────────────────────────────────────────────────────────────
    %% INTEGRATIONS
    %% ─────────────────────────────────────────────────────────────────────
    subgraph OBS["🔌  Integrations"]
        direction LR
        Webhooks["Webhooks\nHMAC-SHA256 signed\ndocument.completed\neval.completed · experiment.winner_found"]
    end

    %% ─────────────────────────────────────────────────────────────────────
    %% CROSS-LAYER CONNECTIONS
    %% ─────────────────────────────────────────────────────────────────────

    %% Auth → query pipeline entry
    Browser -- "authenticated request\naccess_token cookie" --> Gate
    JWTMgr -- "jti revocation\nTTL = refresh lifetime" --> RD
    PG -- "users · api_keys tables" --> AuthSvc
    PG -- "UserSettings\nUserProviderKey" --> UserCfg

    %% Per-user keys flow into pipeline
    UserCfg -- "decrypted key\n→ UserModelContext" --> LLMFact
    UserCfg -- "decrypted embedding key" --> Embedder

    %% Ingestion writes to storage
    Embedder -- "vectors + user_id payload" --> QD
    Embedder -- "BM25 inverted index\nbm25:{user_id}:* keys" --> RD
    FileProc -- "document metadata\nextracted_text" --> PG

    %% Storage feeds query pipeline
    QD -- "top-K semantic results\nuser_id filtered" --> Hybrid
    RD -- "BM25 keyword results" --> Hybrid
    PG -- "system_config overrides\n5-min TTL cache" --> Rewriter

    %% Response feeds evaluation & improvement
    Response -- "async eval trigger\nBackgroundTask" --> EvalOrch
    EvalOrch -- "5 dimension scores\n+ pass/fail flags" --> PG
    EvalOrch -- "event logs" --> MG
    Response -- "user submits feedback\nPOST /feedback" --> Feedback

    %% Improvement loop closes back to pipeline
    PG -- "7-day eval averages\nexperiment results" --> AutoOpt
    SysCfg -- "top_k · alpha\nprompt override" --> Hybrid
    SysCfg -- "system prompt version" --> LLMFact
    ABTest -- "variant config\nrouting by hash" --> Hybrid

    %% Integration taps
    EvalOrch -- "eval.completed event" --> Webhooks
    ABTest -- "experiment.winner_found" --> Webhooks

Module Reference

1. Core Module (`app/core/`)

The foundation of the application providing configuration, logging, and utilities.

File	Purpose
`config.py`	Centralized configuration using Pydantic Settings. Loads from environment variables.
`constants.py`	Application-wide enums and constants (document statuses, strategies, etc.)
`exceptions.py`	Custom exception classes for consistent error handling
`logging.py`	Structured logging with structlog
`security.py`	JWT tokens, password hashing, API key management

Key Classes:

Settings - Configuration container with validation
DocumentStatus, ChunkingStrategy, RetrievalStrategy - Enums for type safety

2. Database Module (`app/db/`)

Database clients for the polyglot persistence architecture.

Database	Purpose	Client
PostgreSQL (`pgvector/pgvector:pg16`)	Documents, chunks, evaluations, experiments, prompts, users/settings, knowledge-graph entities (pgvector embedding column)	SQLAlchemy 2.0 async + asyncpg
MongoDB	Operational event logs (e.g. per-experiment activity timeline). Optional/resilient — degrades gracefully if offline	Motor async
Qdrant	Vector embeddings for semantic search (per-user collections)	qdrant-client
Redis	Response cache, BM25 indices, login rate-limit, JWT jti revocation	redis-py async

PostgreSQL Models:

Document / Chunk - Ingested documents and their chunks
Evaluation - Quality evaluations
Feedback - User feedback
Experiment / ExperimentResult - A/B tests
Prompt / PromptMetric - Prompt versions
User / APIKey - Accounts and API keys; UserSettings / UserProviderKey (auth module)
KnowledgeGraph / KGEntity (pgvector) / KGRelationship / KGEntityMention / KGGraphDocument - Knowledge graphs

The vector extension (pgvector) is enabled by migration 009_knowledge_graph and stores entity embeddings sized by EMBEDDING_DIM (1536 for OpenAI text-embedding-3-small, 768 for Ollama nomic-embed-text).

3. Chunking Module (`app/chunking/`)

Splits documents into optimal chunks for retrieval.

Strategy	Best For	Description
`FixedSizeChunker`	General text	Splits at character boundaries with overlap
`RecursiveChunker`	Structured docs	Preserves paragraphs, sentences, then words
`SemanticChunker`	Long documents	Groups by semantic similarity
`CodeChunker`	Source code	Respects functions, classes, imports

Usage:

from app.chunking.selector import ChunkingStrategySelector

selector = ChunkingStrategySelector()
strategy = selector.select(filename="document.py", content=code)
chunker = selector.get_chunker(strategy)
chunks = await chunker.chunk(text)

4. Embeddings Module (`app/embeddings/`)

Generates vector embeddings for semantic search.

Provider	Model	Dimensions
OpenAI	text-embedding-3-small	1536
OpenAI	text-embedding-3-large	3072
Local	all-MiniLM-L6-v2	384
Local	all-mpnet-base-v2	768
Cohere	embed-english-v3.0	1024

Usage:

from app.embeddings.manager import EmbeddingManager

manager = EmbeddingManager()
embeddings = await manager.embed_batch(["text1", "text2"])
# Returns: List[List[float]]

5. Retrieval Module (`app/retrieval/`)

Multi-strategy retrieval with fusion and reranking.

Component	Description
`VectorRetriever`	Semantic search using Qdrant
`BM25Retriever`	Keyword search with BM25 algorithm
`HybridRetriever`	Combines vector + BM25 results
`ScoreFusion`	RRF or linear weighted fusion
`CrossEncoderReranker`	Neural reranking for precision

Retrieval Flow:

Query → Vector Search ─┐
                       ├─→ Fusion → Reranking → Results
Query → BM25 Search ───┘

Usage:

from app.retrieval.hybrid import HybridRetriever

retriever = HybridRetriever(vector_db, redis_client)
results = await retriever.retrieve(
    query="How does authentication work?",
    top_k=10,
    fusion_method="rrf",
    fusion_weight=0.6,
)

6. LLM Module (`app/llm/`)

Unified interface for multiple LLM providers.

Provider	Models	Features
OpenAI	gpt-4o, gpt-4o-mini	Streaming, function calling
Azure	gpt-4, gpt-35-turbo	Enterprise compliance
Anthropic	claude-3-opus, claude-3-sonnet	Long context
Ollama	llama3, mistral, etc.	Local deployment

Usage:

from app.llm.factory import LLMFactory

llm = LLMFactory.create(provider="openai", model="gpt-4o-mini")
response = await llm.generate(
    prompt="Answer this question...",
    system_prompt="You are a helpful assistant.",
    temperature=0.7,
    max_tokens=500,
)

7. RAG Pipeline (`app/rag/`)

End-to-end retrieval-augmented generation.

Component	Responsibility
`RAGPipeline`	Orchestrates the full RAG flow
`ContextBuilder`	Assembles retrieved chunks into context
`RAGGenerator`	Generates responses with citations

Usage:

from app.rag.pipeline import RAGPipeline

pipeline = RAGPipeline(retriever, generator)
result = await pipeline.query(
    query="What is machine learning?",
    top_k=5,
    include_sources=True,
)
# Returns: RAGResponse with answer and sources

8. Agents Module (`app/agents/`)

AI agents for intelligent operations.

Query Rewriter Agent

Expands and clarifies user queries for better retrieval.

from app.agents.query_rewriter import QueryRewriterAgent

agent = QueryRewriterAgent()
result = await agent.rewrite(
    query="ML models",
    context="Technical documentation",
)
# Returns: rewritten_query, expanded_queries, keywords, intent

Evaluator Agents

Agent	Evaluates	Score Range
`RelevanceEvaluator`	Chunk relevance to query	0.0 - 1.0
`HallucinationEvaluator`	Claims not in context	0.0 - 1.0 (lower is better)
`CitationEvaluator`	Source attribution accuracy	0.0 - 1.0
`QualityEvaluator`	Response quality (clarity, completeness)	0.0 - 1.0
`ContextEvaluator`	Retrieved context quality	0.0 - 1.0

from app.agents.evaluators import HallucinationEvaluator

evaluator = HallucinationEvaluator()
result = await evaluator.evaluate(
    query="What is Python?",
    response="Python is a programming language created in 1991.",
    context_chunks=["Python was created by Guido van Rossum in 1991."],
)
# Returns: hallucination_score, supported_claims, unsupported_claims

Optimizer Agents

Agent	Optimizes	Output
`RetrievalOptimizer`	top_k, fusion weights, strategy	Configuration suggestions
`ChunkingOptimizer`	Chunk size, overlap, strategy	Per-document-type configs
`EmbeddingOptimizer`	Model selection, dimensions	Cost/quality trade-offs
`PromptOptimizer`	Prompt templates	A/B test suggestions
`CostOptimizer`	Resource usage	Savings opportunities

9. Evaluation Engine (`app/evaluation/`)

Automated quality assessment system.

Component	Purpose
`EvaluationEngine`	Orchestrates all evaluations
`MetricsCalculator`	Precision, Recall, MRR, NDCG calculations
`EvaluationReporter`	Generates reports and alerts

Usage:

from app.evaluation.engine import EvaluationEngine

engine = EvaluationEngine(db_session)
suite = await engine.evaluate(
    query="What is RAG?",
    response="RAG stands for...",
    context_chunks=["RAG (Retrieval-Augmented Generation)..."],
    evaluation_types=[EvaluationType.RELEVANCE, EvaluationType.HALLUCINATION],
)
# Returns: EvaluationSuite with overall_score, passed, individual results

10. Optimization Module (`app/optimization/`)

A/B testing and systematic optimization.

Component	Purpose
`ABTestManager`	Manages A/B experiments
`ExperimentFramework`	Statistical experiment infrastructure
`OptimizationOrchestrator`	Coordinates improvement cycles

Usage:

from app.optimization.ab_testing import ABTestManager

manager = ABTestManager(db_session)

# Create experiment
experiment_id = await manager.create_test(
    name="top_k_optimization",
    control_config={"top_k": 5},
    variant_config={"top_k": 10},
    hypothesis="More results improve answer quality",
    traffic_percentage=20.0,
)

# Get variant for request
variant, config = await manager.get_assignment(experiment_id, session_id)

# Record metrics
await manager.record_metric(experiment_id, variant, query_id, {"quality": 0.85})

# Analyze results
results = await manager.get_results(experiment_id)

11. Async Processing & Events

Note: Earlier versions used RabbitMQ + standalone worker processes. V2 removed that layer. The files in app/workers/ are now import-compatibility stubs, and there is no RabbitMQ service or app/events/ publisher. Async work runs in-process:

Async job	Mechanism
Document ingestion (chunk + embed)	FastAPI `BackgroundTasks` (scheduled from the upload endpoint)
Response evaluation (5 dimensions)	FastAPI `BackgroundTask` after each `/search/answer`
Self-improvement / auto-optimization	In-process `asyncio` loop (`AutoOptimizer`, started in the app lifespan)
Knowledge-graph build	FastAPI `BackgroundTasks`
Experiment activity log	Appended to MongoDB `experiment_events` (best-effort)

Outbound webhooks (app/services/webhook.py) fire HMAC-SHA256-signed POSTs on these events: document.completed, evaluation.completed, experiment.winner_found, graph.built.

12. Knowledge Graph (`app/services/knowledge_graph.py`, `app/agents/graph_extraction.py`)

Build user-created, named knowledge graphs from documents and explore them interactively.

Component	Purpose
`KnowledgeGraphService`	Orchestrates an on-demand build (BackgroundTask): gather chunks → extract → resolve → persist
`GraphExtractionAgent`	One LLM call per chunk → entities + relationships (JSON), drops dangling edges/self-loops
Entity resolution	Normalize name, embed via `EmbeddingManager`, merge same-type duplicates by pgvector cosine ≥ `KG_ENTITY_SIM_THRESHOLD`

Build status lifecycle: empty → building → ready / failed. Endpoints under /api/v1/knowledge-graphs (create, assign documents, build, fetch nodes/edges, entity detail, stats). UI at /knowledge-graph.

13. Web UI (`ui/` — Next.js 16)

A TypeScript Next.js 16 app (App Router, Tailwind v4, TanStack Query, @xyflow/react, recharts, lucide-react, sonner).

All API calls go through /api/v1/*, proxied to the backend (no CORS).
Public landing page at / (marketing/overview); unauthenticated users are routed there. After login, the dashboard lives at /dashboard.
Auth screens: /login, /register, /reset-password (password reset — email + new password, no verification; dev/demo only).
Authenticated pages: /dashboard, /search, /documents, /knowledge-graph, /evaluations, /feedback, /experiments, /metrics, /architecture, /settings.
/architecture is an interactive system diagram (React Flow); click any node for a drill-down panel including its effect on the system.

cd ui
npm install
npm run dev      # dev server on :3000 (proxies to backend on :8000)
npm run build    # production build (standalone)

In Docker the UI runs on port 3000.

14. Logging & Metrics

Lightweight, dependency-free operational visibility — no external metrics/tracing infrastructure.

Component	Technology	Purpose
`setup_logging`	Structlog	Structured JSON logs
`CorrelationIdMiddleware`	Custom middleware	Per-request correlation IDs propagated through all log entries
`/api/v1/metrics/*`	Repository queries	JSON summaries: search, quality, feedback, experiments, ingestion

Installation Guide

Prerequisites

Python 3.11 or higher
Docker and Docker Compose
8GB+ RAM recommended

Step 1: Clone Repository

git clone <repository-url>
cd my-rag/backend

Step 2: Create Virtual Environment

python -m venv venv

# Windows
venv\Scripts\activate

# Linux/Mac
source venv/bin/activate

Step 3: Install Dependencies

pip install -e .

# Or with development dependencies
pip install -e ".[dev]"

Step 4: Configure Environment

cp .env.example .env

Edit .env with your settings:

# Required: At least one LLM provider
OPENAI_API_KEY=sk-your-key-here

# Or use Anthropic
ANTHROPIC_API_KEY=sk-ant-your-key

# Or use local Ollama (no key needed)
DEFAULT_LLM_PROVIDER=ollama
DEFAULT_EMBEDDING_PROVIDER=ollama
OLLAMA_HOST=http://localhost:11434   # in Docker: http://host.docker.internal:11434
OLLAMA_MODEL=llama3.1
OLLAMA_EMBEDDING_MODEL=nomic-embed-text
EMBEDDING_DIM=768                     # nomic-embed-text is 768-dim (OpenAI is 1536)

# Required to store per-user provider keys (generate once):
#   python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
ENCRYPTION_KEY=<44-char-fernet-key>

Step 5: Start Infrastructure

docker-compose up -d

This starts:

PostgreSQL + pgvector (port 5432)
MongoDB (port 27017)
Qdrant (port 6333)
Redis (port 6379)
App API (port 8000) and Web UI (port 3000)

RabbitMQ and standalone workers are no longer used (V2 runs async work in-process via FastAPI BackgroundTasks + an in-process optimizer loop).

Step 6: Initialize Databases

python scripts/init_db.py

Step 7: Run Database Migrations

alembic upgrade head

Step 8: (Optional) Seed Test Data

python scripts/seed_data.py

Step 9: Start Application

# Development mode with auto-reload
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

# Production mode
uvicorn app.main:app --host 0.0.0.0 --port 8000 --workers 4

Step 9b: Start the Web UI

cd ui
npm install
npm run dev        # http://localhost:3000 (proxies /api/v1/* to the backend)

Or run the whole stack (backend + UI + all infra) with docker compose up -d — the UI is served on http://localhost:3000.

Step 10: Verify Installation

# Health check
curl http://localhost:8000/health

# Should return: {"status": "healthy", ...}

Access documentation at:

Swagger UI: http://localhost:8000/docs
ReDoc: http://localhost:8000/redoc

Configuration

Environment Variables

Application Settings

Variable	Default	Description
`APP_ENV`	`development`	`development`, `production`
`DEBUG`	`false`	Enable debug mode
`LOG_LEVEL`	`INFO`	`DEBUG`, `INFO`, `WARNING`, `ERROR`
`SECRET_KEY`	(required)	JWT signing key
`ENCRYPTION_KEY`	(required for provider keys)	Fernet key (44-char base64url) used to encrypt per-user provider API keys. Generate with `python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"`
`API_V1_PREFIX`	`/api/v1`	API route prefix

Database Connections

Each DB is configured by host/port/credentials; a full *_URL override is also accepted.

Variable	Default	Description
`POSTGRES_HOST` / `POSTGRES_PORT`	`localhost` / `5432`	PostgreSQL (use the `pgvector/pgvector:pg16` image)
`POSTGRES_USER` / `POSTGRES_PASSWORD` / `POSTGRES_DB`	`raguser` / `ragpassword` / `ragdb`	PostgreSQL credentials
`DATABASE_URL`	(derived)	Optional full SQLAlchemy URL override
`MONGODB_HOST` / `MONGODB_PORT` / `MONGODB_DB`	`localhost` / `27017` / `ragdb`	MongoDB (optional — degrades gracefully if down)
`QDRANT_HOST` / `QDRANT_PORT`	`localhost` / `6333`	Qdrant connection
`QDRANT_COLLECTION`	`documents`	Base Qdrant collection name
`REDIS_HOST` / `REDIS_PORT` / `REDIS_PASSWORD`	`localhost` / `6379` / –	Redis connection

LLM Configuration

Variable	Default	Description
`DEFAULT_LLM_PROVIDER`	`openai`	`openai`, `azure_openai`, `anthropic`, `ollama`
`OPENAI_API_KEY`	-	OpenAI API key
`OPENAI_MODEL`	`gpt-4o`	Default OpenAI model
`ANTHROPIC_API_KEY`	-	Anthropic API key
`ANTHROPIC_MODEL`	`claude-3-5-sonnet-20241022`	Default Anthropic model
`AZURE_OPENAI_ENDPOINT` / `AZURE_OPENAI_API_KEY`	-	Azure OpenAI
`OLLAMA_HOST`	`http://localhost:11434`	Ollama server URL (in Docker: `http://host.docker.internal:11434`)
`OLLAMA_MODEL`	`llama3.1`	Default Ollama model

Embedding Configuration

Variable	Default	Description
`DEFAULT_EMBEDDING_PROVIDER`	`openai`	`openai`, `azure_openai`, `ollama`
`OPENAI_EMBEDDING_MODEL`	`text-embedding-3-small`	OpenAI embedding model
`OLLAMA_EMBEDDING_MODEL`	`nomic-embed-text`	Ollama embedding model
`EMBEDDING_DIM`	`1536`	Vector dimensions — must match the active embedding model (1536 OpenAI, 768 nomic-embed-text) and the Qdrant collection / pgvector column

Retrieval Settings

Variable	Default	Description
`DEFAULT_TOP_K`	`10`	Default candidates to retrieve
`RERANK_TOP_K`	`5`	Results kept after reranking
`HYBRID_ALPHA`	`0.5`	Vector↔BM25 fusion weight (1.0 = pure vector, 0.0 = pure BM25)

Chunking Settings

Variable	Default	Description
`DEFAULT_CHUNK_SIZE`	`512`	Default chunk size
`DEFAULT_CHUNK_OVERLAP`	`50`	Overlap between chunks

Evaluation & Optimization

Variable	Default	Description
`EVALUATION_ENABLED`	`true`	Auto-evaluate responses
`MIN_RELEVANCE_SCORE`	`0.7`	Relevance pass threshold
`MAX_HALLUCINATION_SCORE`	`0.3`	Maximum hallucination rate
`OPTIMIZATION_ENABLED`	`true`	Run the in-process auto-optimizer loop
`OPTIMIZATION_INTERVAL_HOURS`	`24`	How often the optimization cycle runs
`AB_TEST_MIN_SAMPLES`	`50`	Minimum samples before declaring an A/B winner

Auth & Knowledge Graph

Variable	Default	Description
`AUTH_ENABLED`	`true`	Require authentication on protected endpoints
`JWT_EXPIRE_MINUTES` / `REFRESH_TOKEN_EXPIRE_DAYS`	`30` / `7`	Token lifetimes
`REQUIRE_USER_PROVIDER_KEYS`	`false`	If true, users must supply their own provider keys (no env fallback)
`KNOWLEDGE_GRAPH_ENABLED`	`true`	Enable the knowledge-graph feature
`KG_EXTRACTION_MODEL`	`gpt-4o-mini`	LLM used for entity/relationship extraction
`KG_ENTITY_SIM_THRESHOLD`	`0.85`	Cosine threshold for merging duplicate entities
`KG_MAX_CONCURRENCY` / `KG_MAX_CHUNKS_PER_BUILD`	`5` / `2000`	Build concurrency and chunk cap

API Reference

Authentication: When AUTH_ENABLED=true (default), all endpoints except /health and /auth/* require an authenticated user. The UI uses httpOnly access_token cookies; programmatic clients can send Authorization: Bearer <token> or an X-Api-Key header. All data (documents, graphs, experiments) is scoped per user.

Auth API

POST /api/v1/auth/register      # { email, password, full_name? } → sets cookies
POST /api/v1/auth/login         # { email, password } → sets cookies
POST /api/v1/auth/refresh       # rotates tokens (refresh cookie)
POST /api/v1/auth/logout        # revokes refresh jti, clears cookies
GET  /api/v1/auth/me            # current user
POST /api/v1/auth/reset-password   # { email, new_password } — resets password (no verification; dev/demo)

⚠️ reset-password has no email/OTP verification — anyone who knows an account's email can reset it. Fine for dev/demo; not safe for production (use a tokenized email link or OTP there).

Settings API (per-user providers)

GET  /api/v1/settings/providers                 # provider catalog + which keys are set/valid
PUT  /api/v1/settings/providers/{provider}      # store a Fernet-encrypted provider key (+ extra_config, e.g. ollama host)
POST /api/v1/settings/providers/{provider}/verify   # live check the key/connection
PUT  /api/v1/settings/selection                 # choose active llm/embedding provider + model

Documents API

Upload Document

POST /api/v1/documents
Content-Type: multipart/form-data

file: <binary>
title: "Document Title" (optional)
source: "external" (optional)
chunking_strategy: "recursive" (optional)
chunk_size: 512 (optional)
chunk_overlap: 50 (optional)

Response:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "filename": "document.pdf",
  "status": "completed",
  "chunk_count": 45,
  "created_at": "2024-01-15T10:30:00Z"
}

Ingest Text

POST /api/v1/documents/text
Content-Type: application/json

{
  "content": "Your text content here...",
  "title": "Document Title",
  "metadata": {"source": "api"},
  "chunking_strategy": "semantic"
}

List Documents

GET /api/v1/documents?page=1&page_size=20&status=completed

Get Document

GET /api/v1/documents/{document_id}?include_chunks=true

Delete Document

DELETE /api/v1/documents/{document_id}

Search API

Search Chunks

POST /api/v1/search
Content-Type: application/json

{
  "query": "How does authentication work?",
  "top_k": 10,
  "strategy": "hybrid",
  "rerank": true,
  "filter_document_ids": ["uuid1", "uuid2"],
  "min_score": 0.5,
  "include_content": true
}

Response:

{
  "query": "How does authentication work?",
  "results": [
    {
      "chunk_id": "...",
      "document_id": "...",
      "document_title": "Auth Guide",
      "content": "Authentication is handled by...",
      "score": 0.92,
      "rank": 1,
      "metadata": {}
    }
  ],
  "total_results": 10,
  "strategy_used": "hybrid",
  "search_time_ms": 45.2,
  "reranked": true
}

Generate Answer (RAG)

POST /api/v1/search/answer
Content-Type: application/json

{
  "query": "How does authentication work?",
  "top_k": 5,
  "rerank": true,
  "system_prompt": "You are a technical documentation assistant.",
  "max_tokens": 500,
  "temperature": 0.7,
  "include_sources": true
}

Response:

{
  "query": "How does authentication work?",
  "answer": "Authentication in this system works through JWT tokens...",
  "sources": [
    {
      "document_id": "...",
      "document_title": "Auth Guide",
      "chunk_id": "...",
      "chunk_index": 3,
      "content_preview": "JWT tokens are issued...",
      "relevance_score": 0.92
    }
  ],
  "model_used": "gpt-4o-mini",
  "search_time_ms": 45,
  "generation_time_ms": 1200,
  "total_time_ms": 1245,
  "query_id": "q-123456"
}

Stream Answer

POST /api/v1/search/answer/stream
Content-Type: application/json

{
  "query": "Explain the architecture",
  "top_k": 5
}

Returns: text/plain stream with generated tokens.

Feedback API

Submit Feedback

POST /api/v1/feedback
Content-Type: application/json

{
  "query_id": "q-123456",
  "feedback_type": "rating",
  "rating": 4,
  "comment": "Good answer but could be more detailed",
  "session_id": "session-abc"
}

Feedback Types:

thumbs_up / thumbs_down - Quick binary feedback
rating - 1-5 star rating
correction - User provides correct answer
detailed - Full feedback with comments

Quick Feedback

POST /api/v1/feedback/quick
Content-Type: application/json

{
  "query_id": "q-123456",
  "is_positive": true,
  "session_id": "session-abc"
}

Get Feedback Aggregation

GET /api/v1/feedback/aggregation?start_date=2024-01-01&end_date=2024-01-31

Response:

{
  "total_feedback": 1500,
  "thumbs_up_count": 1200,
  "thumbs_down_count": 300,
  "thumbs_up_rate": 0.8,
  "average_rating": 4.2,
  "rating_distribution": {"1": 50, "2": 75, "3": 150, "4": 400, "5": 500},
  "correction_count": 25,
  "satisfaction_rate": 0.85
}

Evaluations API

List Evaluations

GET /api/v1/evaluations?evaluation_type=relevance&min_score=0.5

Get Evaluations for Query

GET /api/v1/evaluations/query/{query_id}

Response:

{
  "query_id": "q-123456",
  "evaluations": [
    {
      "type": "relevance",
      "score": 0.85,
      "passed": true,
      "details": {...}
    },
    {
      "type": "hallucination",
      "score": 0.95,
      "passed": true,
      "details": {...}
    }
  ],
  "aggregated_scores": {
    "relevance": 0.85,
    "hallucination": 0.95,
    "overall": 0.90
  }
}

Get Evaluation Summary

GET /api/v1/evaluations/summary?start_date=2024-01-01

Experiments API

Create Experiment

POST /api/v1/experiments
Content-Type: application/json

{
  "name": "top_k_optimization",
  "description": "Test if more results improve quality",
  "hypothesis": "Increasing top_k from 5 to 10 improves answer quality",
  "control_config": {"top_k": 5, "rerank": true},
  "variant_config": {"top_k": 10, "rerank": true},
  "traffic_percentage": 20.0,
  "min_samples": 100
}

Start Experiment

POST /api/v1/experiments/{experiment_id}/start

Get Variant Assignment

POST /api/v1/experiments/{experiment_id}/assign?session_id=abc123

Response:

{
  "experiment_id": "...",
  "variant": "variant",
  "config": {"top_k": 10, "rerank": true},
  "session_id": "abc123"
}

Record Result

POST /api/v1/experiments/{experiment_id}/results
Content-Type: application/json

{
  "variant": "variant",
  "query_id": "q-123456",
  "metrics": {
    "relevance_score": 0.88,
    "quality_score": 0.85,
    "latency_ms": 150
  }
}

Get Analysis

GET /api/v1/experiments/{experiment_id}/analysis

Response:

{
  "experiment_id": "...",
  "status": "running",
  "control_metrics": {
    "relevance_score": {"mean": 0.82, "count": 500},
    "quality_score": {"mean": 0.80, "count": 500}
  },
  "variant_metrics": {
    "relevance_score": {"mean": 0.88, "count": 120},
    "quality_score": {"mean": 0.85, "count": 120}
  },
  "analysis": {
    "relative_improvements": {
      "relevance_score": 7.3,
      "quality_score": 6.2
    },
    "winner": "variant",
    "confidence_level": 0.95,
    "recommendation": "Results are statistically significant"
  }
}

Get Activity (live steps + progress + event log)

GET /api/v1/experiments/{experiment_id}/events

Returns the experiment's lifecycle steps (created → started → collecting → enough-data → completed), progress (per-variant sample counts vs min_samples), and the event timeline from MongoDB. Note: an experiment is passive — it only collects samples when real /search/answer queries run while it is running.

{
  "experiment_id": "...", "name": "test", "status": "running",
  "logging_available": true,
  "steps": [{"key": "collecting", "label": "Collecting samples", "status": "active", "detail": "12 / 100 samples"}],
  "progress": {"control": 6, "variant": 6, "total": 12, "min_samples": 100, "percent": 12.0},
  "events": [{"ts": "...", "type": "result_recorded", "level": "info", "message": "Query routed to 'variant' variant · 240ms · 5 sources"}]
}

Knowledge Graph API

POST   /api/v1/knowledge-graphs                          # create { name, description?, document_ids? }
GET    /api/v1/knowledge-graphs                          # list the user's graphs
GET    /api/v1/knowledge-graphs/{id}                     # graph metadata + status
DELETE /api/v1/knowledge-graphs/{id}
GET    /api/v1/knowledge-graphs/{id}/documents           # assigned documents
PUT    /api/v1/knowledge-graphs/{id}/documents           # { document_ids } — replace assignment
POST   /api/v1/knowledge-graphs/{id}/build               # kick off a background build
GET    /api/v1/knowledge-graphs/{id}/data?type=&search=&limit=   # nodes + edges for visualization
GET    /api/v1/knowledge-graphs/{id}/entities/{entity_id}        # entity detail: description, neighbors, source chunks
GET    /api/v1/knowledge-graphs/{id}/stats               # counts by entity type

Build flow: create a graph → assign documents → build → poll GET /{id} until status: ready → render /{id}/data and click a node for /{id}/entities/{entity_id}.

Metrics API

Get System Metrics

GET /api/v1/metrics?start_date=2024-01-01&end_date=2024-01-31

Response:

{
  "period": {"start": "...", "end": "..."},
  "documents": {"total": 500, "chunks": 25000},
  "evaluations": {"total": 10000},
  "feedback": {
    "total": 5000,
    "positive_rate": 0.85,
    "average_rating": 4.2
  },
  "experiments": {"total": 10, "active": 2},
  "vector_db": {"vectors_count": 25000}
}

Usage Guide

Basic RAG Query

import httpx

async def ask_question(query: str):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:8000/api/v1/search/answer",
            json={
                "query": query,
                "top_k": 5,
                "rerank": True,
                "include_sources": True,
            }
        )
        return response.json()

# Example
result = await ask_question("How do I configure authentication?")
print(f"Answer: {result['answer']}")
print(f"Sources: {result['sources']}")

Document Ingestion

import httpx

async def upload_document(file_path: str):
    async with httpx.AsyncClient() as client:
        with open(file_path, "rb") as f:
            response = await client.post(
                "http://localhost:8000/api/v1/documents",
                files={"file": f},
                data={
                    "title": "My Document",
                    "chunking_strategy": "recursive",
                }
            )
        return response.json()

# Check status
async def check_status(document_id: str):
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"http://localhost:8000/api/v1/documents/{document_id}/status"
        )
        return response.json()

Streaming Responses

import httpx

async def stream_answer(query: str):
    async with httpx.AsyncClient() as client:
        async with client.stream(
            "POST",
            "http://localhost:8000/api/v1/search/answer/stream",
            json={"query": query, "top_k": 5}
        ) as response:
            async for chunk in response.aiter_text():
                print(chunk, end="", flush=True)

Collecting Feedback

import httpx

async def submit_feedback(query_id: str, is_positive: bool):
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "http://localhost:8000/api/v1/feedback/quick",
            json={
                "query_id": query_id,
                "is_positive": is_positive,
            }
        )
        return response.json()

Running A/B Tests

import httpx

class ABTestClient:
    def __init__(self, base_url: str):
        self.base_url = base_url
        self.client = httpx.AsyncClient()

    async def create_experiment(self, name: str, control: dict, variant: dict):
        response = await self.client.post(
            f"{self.base_url}/api/v1/experiments",
            json={
                "name": name,
                "control_config": control,
                "variant_config": variant,
                "traffic_percentage": 20.0,
                "min_samples": 100,
            }
        )
        return response.json()["id"]

    async def get_variant(self, experiment_id: str, session_id: str):
        response = await self.client.post(
            f"{self.base_url}/api/v1/experiments/{experiment_id}/assign",
            params={"session_id": session_id}
        )
        data = response.json()
        return data["variant"], data["config"]

    async def record_result(self, experiment_id: str, variant: str,
                           query_id: str, metrics: dict):
        await self.client.post(
            f"{self.base_url}/api/v1/experiments/{experiment_id}/results",
            json={
                "variant": variant,
                "query_id": query_id,
                "metrics": metrics,
            }
        )

Self-Improvement System

The system continuously improves through an automated cycle:

1. Evaluation Phase

Every query-response pair is automatically evaluated:

Query + Response + Context
            │
            ▼
┌─────────────────────────────────────┐
│         Evaluation Engine           │
├─────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  │
│  │  Relevance  │  │Hallucination│  │
│  │  Evaluator  │  │  Evaluator  │  │
│  └─────────────┘  └─────────────┘  │
│  ┌─────────────┐  ┌─────────────┐  │
│  │  Citation   │  │   Quality   │  │
│  │  Evaluator  │  │  Evaluator  │  │
│  └─────────────┘  └─────────────┘  │
└─────────────────────────────────────┘
            │
            ▼
    Evaluation Scores + Alerts

2. Feedback Collection

User feedback is collected and analyzed:

Thumbs up/down: Quick signal
Ratings: 1-5 star scores
Corrections: User-provided correct answers
Comments: Detailed feedback

3. Optimization Analysis

AI agents analyze performance data:

# Triggered periodically or manually
from app.pipelines.optimization import OptimizationPipeline

pipeline = OptimizationPipeline(db_session)
report = await pipeline.run_full_analysis()

for suggestion in report.suggestions:
    print(f"{suggestion.category}: {suggestion.parameter}")
    print(f"  Current: {suggestion.current_value}")
    print(f"  Suggested: {suggestion.suggested_value}")
    print(f"  Expected Improvement: {suggestion.expected_improvement:.1%}")

4. A/B Testing

Suggestions are tested systematically:

Suggestion: Increase top_k from 5 to 10
                    │
                    ▼
         ┌──────────────────┐
         │   A/B Test       │
         │  20% traffic     │
         └──────────────────┘
                    │
        ┌───────────┴───────────┐
        ▼                       ▼
   ┌─────────┐             ┌─────────┐
   │ Control │             │ Variant │
   │ top_k=5 │             │top_k=10 │
   └─────────┘             └─────────┘
        │                       │
        └───────────┬───────────┘
                    ▼
         Statistical Analysis
                    │
                    ▼
         Winner Determination

5. Automated Rollout

Winning configurations are applied:

from app.optimization.orchestrator import OptimizationOrchestrator

orchestrator = OptimizationOrchestrator(db_session)

# Run optimization cycle
report = await orchestrator.run_optimization_cycle()

# Check status
status = await orchestrator.get_status()
print(f"Active experiments: {status['active_experiments']}")

Monitoring & Observability

JSON Metrics Endpoints

The API exposes repository-backed metrics summaries (no external scraper required):

Endpoint	Description
`GET /api/v1/metrics`	System overview: documents, evaluations, feedback, experiments, vectors
`GET /api/v1/metrics/search`	Search volume and latency
`GET /api/v1/metrics/quality`	Evaluation score trends
`GET /api/v1/metrics/feedback`	Feedback analytics
`GET /api/v1/metrics/experiments`	A/B experiment results
`GET /api/v1/metrics/ingestion`	Document ingestion stats

Structured Logging

Logs are JSON-formatted for easy parsing:

{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "info",
  "logger": "app.rag.pipeline",
  "message": "RAG query completed",
  "correlation_id": "abc-123",
  "query_id": "q-456",
  "duration_ms": 1250,
  "chunks_retrieved": 5,
  "model": "gpt-4o-mini"
}

Development Guide

Running Tests

# All tests
pytest tests/ -v

# Unit tests only
pytest tests/unit/ -v

# Integration tests
pytest tests/integration/ -v

# With coverage
pytest tests/ -v --cov=app --cov-report=html

Code Style

# Format code
black app/ tests/
isort app/ tests/

# Type checking
mypy app/

# Linting
ruff app/

Adding a New Evaluator

Create evaluator in app/agents/evaluators/:

from app.agents.base import BaseAgent, AgentConfig

class MyEvaluator(BaseAgent[MyInput, MyOutput]):
    def _default_config(self) -> AgentConfig:
        return AgentConfig(
            name="my_evaluator",
            description="Evaluates something specific",
            model="gpt-4o-mini",
        )

    def _build_prompt(self, input_data: MyInput) -> str:
        return f"Evaluate this: {input_data.content}"

    def _parse_response(self, response: str) -> MyOutput:
        # Parse LLM response
        pass

    async def _execute(self, input_data: MyInput) -> MyOutput:
        prompt = self._build_prompt(input_data)
        response = await self._call_llm(prompt)
        return self._parse_response(response)

Register in app/agents/evaluators/__init__.py
Add to EvaluationEngine if needed

Adding a New LLM Provider

Create provider in app/llm/:

from app.llm.base import BaseLLM, LLMResponse

class MyLLM(BaseLLM):
    async def generate(
        self,
        prompt: str,
        system_prompt: Optional[str] = None,
        **kwargs,
    ) -> LLMResponse:
        # Implementation
        pass

    async def generate_stream(self, ...):
        # Streaming implementation
        pass

Register in app/llm/factory.py:

@staticmethod
def create(provider: str, **kwargs) -> BaseLLM:
    if provider == "myprovider":
        return MyLLM(**kwargs)

Database Migrations

# Create migration
alembic revision --autogenerate -m "Add new table"

# Apply migrations
alembic upgrade head

# Rollback
alembic downgrade -1

Troubleshooting

Common Issues

Connection Refused to Services

# Check service status
docker-compose ps

# View logs
docker-compose logs postgres
docker-compose logs qdrant

# Restart services
docker-compose restart

Out of Memory (Qdrant)

Increase Qdrant memory in docker-compose.yml:

qdrant:
  deploy:
    resources:
      limits:
        memory: 4G

Slow Embedding Generation

Enable embedding cache in Redis
Use smaller embedding model
Increase batch size

# In config
EMBEDDING_CACHE_ENABLED=true
EMBEDDING_BATCH_SIZE=100

High LLM Costs

Use gpt-4o-mini for evaluations
Enable response caching
Reduce max_tokens
Use local Ollama for non-critical tasks

Evaluation Timeouts

Increase timeout in agent config:

AgentConfig(
    timeout=60.0,  # seconds
    retry_attempts=3,
)

Debug Mode

Enable debug logging:

LOG_LEVEL=DEBUG
DEBUG=true

Health Check Endpoints

# Basic health
curl http://localhost:8000/health

# Readiness (all dependencies)
curl http://localhost:8000/health/ready

# Detailed component status
curl http://localhost:8000/health/components

License

MIT License - see LICENSE file for details.

Support

Issues: Submit via GitHub Issues
Documentation: This README and /docs endpoint
API Reference: Swagger UI at /docs

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
alembic		alembic
app		app
docker		docker
scripts		scripts
tests		tests
ui		ui
.env.example		.env.example
.gitignore		.gitignore
DEPLOY_RAILWAY.md		DEPLOY_RAILWAY.md
README.md		README.md
alembic.ini		alembic.ini
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
test_api.py		test_api.py
test_response.py		test_response.py
test_text.py		test_text.py

Folders and files

Latest commit

History

Repository files navigation

Cognesys — a self-improving RAG platform

Table of Contents

Overview

Key Features

Architecture

Module Reference

1. Core Module (app/core/)

2. Database Module (app/db/)

3. Chunking Module (app/chunking/)

4. Embeddings Module (app/embeddings/)

5. Retrieval Module (app/retrieval/)

6. LLM Module (app/llm/)

7. RAG Pipeline (app/rag/)

8. Agents Module (app/agents/)

Query Rewriter Agent

Evaluator Agents

Optimizer Agents

9. Evaluation Engine (app/evaluation/)

10. Optimization Module (app/optimization/)

11. Async Processing & Events

12. Knowledge Graph (app/services/knowledge_graph.py, app/agents/graph_extraction.py)

13. Web UI (ui/ — Next.js 16)

14. Logging & Metrics

Installation Guide

Prerequisites

Step 1: Clone Repository

Step 2: Create Virtual Environment

Step 3: Install Dependencies

Step 4: Configure Environment

Step 5: Start Infrastructure

Step 6: Initialize Databases

Step 7: Run Database Migrations

Step 8: (Optional) Seed Test Data

Step 9: Start Application

Step 9b: Start the Web UI

Step 10: Verify Installation

Configuration

Environment Variables

Application Settings

Database Connections

LLM Configuration

Embedding Configuration

Retrieval Settings

Chunking Settings

Evaluation & Optimization

Auth & Knowledge Graph

API Reference

Auth API

Settings API (per-user providers)

Documents API

Upload Document

Ingest Text

List Documents

Get Document

Delete Document

Search API

Search Chunks

Generate Answer (RAG)

Stream Answer

Feedback API

Submit Feedback

Quick Feedback

Get Feedback Aggregation

Evaluations API

List Evaluations

Get Evaluations for Query

Get Evaluation Summary

Experiments API

Create Experiment

Start Experiment

Get Variant Assignment

Record Result

Get Analysis

Get Activity (live steps + progress + event log)

Knowledge Graph API

Metrics API

1. Core Module (`app/core/`)

2. Database Module (`app/db/`)

3. Chunking Module (`app/chunking/`)

4. Embeddings Module (`app/embeddings/`)

5. Retrieval Module (`app/retrieval/`)

6. LLM Module (`app/llm/`)

7. RAG Pipeline (`app/rag/`)

8. Agents Module (`app/agents/`)

9. Evaluation Engine (`app/evaluation/`)

10. Optimization Module (`app/optimization/`)

12. Knowledge Graph (`app/services/knowledge_graph.py`, `app/agents/graph_extraction.py`)

13. Web UI (`ui/` — Next.js 16)

Packages