Tails

Datadog RAG, built in Rust.

Includes a one-shot Datadog indexer, RAG API, and CLI with automatic inference of service and environment. The server dynamically selects topK based on question intent.

---

Quickstart

# 1. Index Datadog data into Qdrant
cd crates/rag-indexer
cargo run

# 2. Start the RAG API
cd ../rag-api
cargo run

# 3. Ask a question via CLI
cd ../rag-cli
RAG_API_BASE=http://localhost:5191 cargo run -- ask "why did auth-api spike yesterday?"

The planner infers service/environment automatically.
The server chooses K dynamically — no --k flag needed.

---

Crates Overview

Crate	Description
rag-core	Domain models, OpenAI, Qdrant (search + upsert), Datadog client (monitors, incidents, logs, dashboards, metrics, SLOs), chunker, planner, reranker, RAG service.
rag-api	Axum REST API — /ask/plan (intent + inferred filters) and /ask (retrieval + answer).
rag-cli	CLI that calls the API. Planner infers service/env; server decides top-K.
rag-indexer	One-shot indexer with watermark for Datadog → Qdrant. Perfect for Kubernetes CronJob.

---

Development

Prerequisites

• Rust (latest stable)
• Docker (for containerization)
• NASM (for cryptographic aws_lc_rs feature) - NO LONGER REQUIRED, using native-tls instead, leaving this here in case I decide to move back to rustls in the future for better performance
• CMake (for cryptographic aws_lc_rs feature) - NO LONGER REQUIRED, using native-tls instead, leaving this here in case I decide to move back to rustls in the future for better performance

Environment Variables

# OpenAI
OPENAI_API_KEY=...
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_CHAT_MODEL=o4-mini

# Qdrant
QDRANT_ENDPOINT=http://qdrant:6333
QDRANT_COLLECTION=datadog_rag

# Datadog
DD_API_KEY=...
DD_APP_KEY=...
DD_SITE=datadoghq.eu    # or datadoghq.com

# Indexer
INDEXER_WATERMARK=/data/watermark.json
INDEXER_LOOKBACK_MINUTES=90

# Retrieval tuning (optional)
RAG_TOPK_DEFAULT=16
RAG_TOPK_MAX=32
RAG_SEARCH_CANDIDATES=64

---

Running

API

cd crates/rag-api
cargo run

CLI

Quick Install (recommended):

Linux/macOS:

curl -fsSL https://raw.githubusercontent.com/MattiasHognas/tails/main/install.sh | bash

Windows (PowerShell):

irm https://raw.githubusercontent.com/MattiasHognas/tails/main/install.ps1 | iex

The installer will:

• Detect your platform automatically
• Download the latest release
• Update existing installation if found
• Add to PATH (on Windows)

Advanced Installation Options:

Install to a custom directory (Linux/macOS):

INSTALL_DIR=/usr/local/bin VERSION=v1.0.0 bash install.sh

Install to a custom directory (Windows):

.\install.ps1 -InstallDir "C:\Tools\rag-cli" -Version "v1.0.0"

Default installation locations:

• Linux/macOS: $HOME/.local/bin/rag-cli
• Windows: %LOCALAPPDATA%\rag-cli\rag-cli.exe

Manual Download:

• Linux (x86_64): rag-cli-linux-x86_64
• Linux (aarch64): rag-cli-linux-aarch64
• macOS (Intel): rag-cli-macos-x86_64
• macOS (Apple Silicon): rag-cli-macos-aarch64
• Windows (x86_64): rag-cli-windows-x86_64.exe

Binaries are automatically built and available as GitHub Release artifacts.

Build from source (development)

cd crates/rag-cli
RAG_API_BASE=http://localhost:5191 cargo run -- ask "why did auth-api spike yesterday?"

No --k flag — server dynamically chooses K.
No --env or --service needed — planner infers them automatically (e.g., “auth-api prod”).

Manual override if desired:

cargo run -- ask "auth-api latency spikes" --env prod --service auth-api

Indexer (manual run)

cd crates/rag-indexer
INDEXER_WATERMARK=./watermark.json DD_API_KEY=... DD_APP_KEY=... DD_SITE=datadoghq.eu OPENAI_API_KEY=... QDRANT_ENDPOINT=http://localhost:6333 QDRANT_COLLECTION=datadog_rag cargo run

---

Docker

Building Images

Build server services from pre-compiled binaries:

# Build release binaries first
cargo build --release

# Build Docker images
docker build -f crates/rag-indexer/Dockerfile -t rag-indexer:latest .
docker build -f crates/rag-api/Dockerfile -t rag-api:latest .

Running with Docker

Indexer:

docker run --rm \
  -e OPENAI_API_KEY=... \
  -e QDRANT_ENDPOINT=http://qdrant:6333 \
  -e QDRANT_COLLECTION=datadog_rag \
  -e DD_API_KEY=... \
  -e DD_APP_KEY=... \
  -e DD_SITE=datadoghq.eu \
  rag-indexer:latest

API:

docker run --rm -p 5191:5191 \
  -e OPENAI_API_KEY=... \
  -e QDRANT_ENDPOINT=http://qdrant:6333 \
  -e QDRANT_COLLECTION=datadog_rag \
  -e DD_API_KEY=... \
  -e DD_APP_KEY=... \
  -e DD_SITE=datadoghq.eu \
  rag-api:latest

---

Example CronJob (Kubernetes)

apiVersion: batch/v1
kind: CronJob
metadata:
  name: rag-indexer
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: indexer
            image: ghcr.io/yourorg/rag-indexer:latest
            env:
            - name: OPENAI_API_KEY
              valueFrom: { secretKeyRef: { name: openai, key: apiKey } }
            - name: QDRANT_ENDPOINT
              value: http://qdrant:6333
            - name: QDRANT_COLLECTION
              value: datadog_rag
            - name: DD_API_KEY
              valueFrom: { secretKeyRef: { name: datadog, key: apiKey } }
            - name: DD_APP_KEY
              valueFrom: { secretKeyRef: { name: datadog, key: appKey } }
            - name: DD_SITE
              value: datadoghq.eu
            - name: INDEXER_WATERMARK
              value: /data/watermark.json
            - name: INDEXER_LOOKBACK_MINUTES
              value: "90"
            volumeMounts:
            - name: data
              mountPath: /data
          volumes:
          - name: data
            emptyDir: {}

---

Testing

Running Tests

# Run all tests
cargo test

# Run tests for a specific crate
cargo test -p rag-core

# Run tests with output
cargo test -- --nocapture

Mutation Testing

The project uses cargo-mutants for mutation testing to identify missing test coverage:

# Install cargo-mutants
cargo install cargo-mutants

# Run mutation tests
cargo mutants

# Run on specific package
cargo mutants --package rag-core

For detailed mutation testing results and recommendations, see MUTATION_TESTING_REPORT.md.

Current Test Coverage:

• 141 unit tests covering core functionality, API logic, and indexer
• Mutation testing: 43.1% caught (62/144 mutants) - +24.7% improvement!
• Strong coverage of OpenAI client (100%), Qdrant client (80%), and choose_topk logic (76.5%)
• See report for areas needing additional test coverage

---

Notes

• The planner (/ask/plan) extracts intent, time window, service/env, and clarifying questions.
• The API embeds the rewritten query, searches Qdrant, and reranks results using hybrid heuristics.
• The CLI automatically displays clarifying questions if planner uncertainty is high.
• Default values can be tuned via environment variables on the API service.