diff --git a/AGENTS.md b/AGENTS.md index dd6daf2..0f0a78b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,9 +17,9 @@ pnpm inspector ## Safety rules -- Never log, print, or commit `SUPABASE_SERVICE_KEY` or `API_BEARER_TOKEN`. -- Keep server-only secrets server-side. Do not place service-role credentials in desktop renderer code, website bundles, or client config files. -- Treat desktop distribution as client-like: only call server APIs from the desktop app, never embed privileged Supabase keys. +- Never log, print, or commit `DATABASE_URL` or `API_BEARER_TOKEN`. +- Keep server-only secrets server-side. Do not place database credentials in desktop renderer code, website bundles, or client config files. +- Treat desktop distribution as client-like: only call server APIs from the desktop app, never embed privileged database credentials. ## MCP tool inventory diff --git a/CHANGELOG.md b/CHANGELOG.md index 24881b9..d707e1f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,52 @@ All notable changes to Textrawl are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [0.4.0] - 2026-06-13 + +### Added + +- **Large-upload pipeline**: GCS-backed resumable upload sessions (server initiates the session, the browser PUTs directly to the returned URI) with `/api/upload/init` and `/api/upload/complete`, an idempotent upload state machine, and the `uploads` / `upload_entries` metadata schema (`scripts/setup-db-uploads.sql`) +- **Cloud Tasks processing**: background processing queue with OIDC authentication and dedupe, plus an internal Cloud Run processing endpoint for async single-file ingestion +- **Processor handler registry**: pluggable converter registry covering Tier-1 file types (pdf, docx, html, xlsx, csv, json, text) with safe streaming ZIP extraction +- **Dashboard resumable upload UX**: large-upload flow that resolves at `queued` so processing runs in the background, an honest file-type support matrix, and clear structured upload errors +- **`scan` and `split` CLI commands**: analyze converted markdown for upload readiness and split oversized files at heading boundaries before upload +- **Social export converters**: Spotify, Reddit, Facebook, and Instagram data-export converters +- **Optional Redis-backed rate limiting**: set `REDIS_URL` to share rate-limit counters across server instances (falls back to in-memory) +- **`health_check` MCP tool**: per-component diagnostic (database, documents, chunks, memory, conversations, insights) — tool count is now 26 + +### Changed + +- **Google AI embedding default** → `gemini-embedding-2-preview` (3072 dimensions). **Breaking**: changing from the previous `text-embedding-004` (768d) requires re-embedding all documents with `scripts/setup-db-google.sql` +- **Default `EXTRACTION_MODEL`** → `claude-haiku-4-5-20251001` +- **Provider resilience**: retry transient embedding/Anthropic failures with exponential backoff; Ollama fetch timeout with typed provider errors +- **Bounded ingestion memory**: cap memory across the upload→process pipeline and defer per-document memory extraction off the processing critical path + +### Fixed + +- Chunker O(n) force-split to stop OOM on large single-paragraph text +- Keep OpenAI embedding batches under the 300K-token request cap +- Tolerate truncated/fenced LLM JSON during memory extraction; raise extraction `max_tokens` +- Return structured JSON for Multer/413/file-type upload errors; wire `MAX_SINGLE_FILE_SIZE_MB` into the multer limit +- Dashboard: derive the WebSocket URL from the API base (no localhost fallback in production), route ZIP uploads through the resumable path, and suppress the React #418 hydration warning +- Move `unzipper` to runtime dependencies so ZIP processing boots on Cloud Run +- Remove a dead xlsx budget assignment flagged by CodeQL + +### Documentation + +- Synced docs to code: replaced Supabase setup with Neon / `DATABASE_URL` across installation, configuration, RUNBOOK, troubleshooting, Docker and Cloud Run guides, and per-tool error tables +- Corrected the Google AI embedding model and dimensions (3072d) across all documentation +- Added the `health_check` tool reference and updated the tool count from 25 to 26 +- Added `scan` and `split` CLI reference pages and documented the Spotify/Reddit/Facebook/Instagram converters +- Added an Agent-to-Agent (A2A) protocol reference page +- Archived completed implementation plans (`docs/plans/`) and the improvement tracker to `docs/archive/` + +### Dependencies + +- Add `@google-cloud/storage` for GCS resumable uploads +- Add `@google-cloud/tasks` and `google-auth-library` for Cloud Tasks queueing and OIDC +- Add `redis` and `rate-limit-redis` for optional shared rate limiting +- Move `unzipper` to runtime dependencies + ## [0.3.0] - 2026-03-26 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 793025f..66e9f66 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -90,7 +90,7 @@ Implementations: `src/tools/*.ts`. Full descriptions and schemas in `README.md`. ## Postgres analysis tools -- Gated on `DATABASE_URL` env var (direct `pg` connection, independent of Supabase) +- Gated on `DATABASE_URL` env var (direct `pg` connection, independent of the main database client) - MCP tools: `src/tools/pg-analyze.ts` - Analysis engine: `src/services/pg-analyze/` - CLI: `pnpm pg:analyze` (`scripts/cli/pg-analyze.ts`) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b366ee5..0a33d57 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,7 +8,7 @@ Thanks for your interest in contributing! This guide will help you get started. - Node.js >= 22.0.0 - Docker (for local database) -- A Supabase account OR local PostgreSQL with pgvector +- A Neon account (or any pgvector-enabled Postgres) OR local PostgreSQL with pgvector - OpenAI API key OR [Ollama](https://ollama.com) (free, local) for embeddings ### Quick Start @@ -24,7 +24,7 @@ pnpm install # Run the setup script (generates .env with secure token) pnpm setup -# Start local database (optional - or use Supabase) +# Start local database (optional - or use a hosted Postgres like Neon) docker-compose -f docker-compose.local.yml up -d postgres # Initialize the database @@ -44,7 +44,7 @@ pnpm inspector # Test with MCP Inspector # File conversion (see docs/cli/) pnpm convert # Convert files (mbox, eml, html, takeout) -pnpm upload # Upload converted markdown to Supabase +pnpm upload # Upload converted markdown to your knowledge base pnpm ui # Web UI for conversion ``` @@ -107,7 +107,7 @@ server.tool('tool_name', { ```text src/ ├── api/ # Express routes and middleware -├── db/ # Database queries (Supabase) +├── db/ # Database queries (Postgres) ├── services/ # Business logic (embeddings, chunking) ├── tools/ # MCP tool definitions ├── types/ # TypeScript type definitions diff --git a/TODO.md b/TODO.md index 7aae1d8..7960ba0 100644 --- a/TODO.md +++ b/TODO.md @@ -1,162 +1,6 @@ -# Docs Sync TODO +# Roadmap TODO -Audit completed 2026-05-01. Six discrepancies found between `/docs` and the codebase. Each phase below fits within a 200k context window session. - ---- - -## Phase 1 — Database provider: Supabase → Neon - -The project migrated to Neon PostgreSQL (`DATABASE_URL`), but docs still instruct users to create a Supabase project and set `SUPABASE_URL`/`SUPABASE_SERVICE_KEY`. - -### Files to update - -**`docs/getting-started/installation.mdx`** - -- Lines 66–135: Replace "Option 1: Supabase (Recommended)" section with Neon setup (create project at neon.tech, copy pooled connection string into `DATABASE_URL`) -- Lines 128–135: Replace required vars block (`SUPABASE_URL`, `SUPABASE_SERVICE_KEY`) with `DATABASE_URL=postgresql://...` and `OPENAI_API_KEY=sk-...` - -**`docs/getting-started/configuration.mdx`** - -- Lines 11–18: Rename section header "Database Connection"; replace `SUPABASE_URL`/`SUPABASE_SERVICE_KEY` rows with `DATABASE_URL` (required, Neon pooled connection string) -- Lines 224–256: Update all three example configs (Dev, Production, Self-hosted) to use `DATABASE_URL` instead of the two Supabase vars - -**`docs/cli/index.mdx`** - -- Line 15: Change "Upload to Supabase" → "Upload to knowledge base" -- Line 73: Change "Supabase Knowledge Base" → "Knowledge Base" in the workflow diagram - -**`docs/architecture/embeddings.mdx`** - -- Line 32: Change "Run in Supabase SQL Editor" → `psql $DATABASE_URL -f scripts/setup-db.sql` -- Line 48: Same change for the Google AI schema instruction - -**`docs/mcp-tools/index.mdx`** - -- Line 349: Error table row "Database not configured" — change fix text from "Set `SUPABASE_URL` and `SUPABASE_SERVICE_KEY`" → "Set `DATABASE_URL`" -- Line 160: pg-analyze section — change "independent of Supabase client" → "independent of the main database client" - -**`docs/guides/supabase-requirements.mdx`** - -- Retitle to "Database Requirements" (or rename file to `database-requirements.mdx` and update `meta.json`) -- Replace Supabase-specific tier names/links with Neon equivalent (Neon free tier, Neon Pro, etc.) -- Keep pgvector/HNSW sizing math — it applies to any Postgres deployment -- Update any "Supabase Dashboard" UI references to generic psql/Neon console instructions - -**`docs/RUNBOOK.md`** - -- Update "Supabase setup" section heading and profile descriptions to reference Neon -- Replace `SUPABASE_URL`/`SUPABASE_SERVICE_KEY` wherever they appear with `DATABASE_URL` - ---- - -## Phase 2 — Google AI embeddings model/dimensions + `health_check` tool - -Two independent issues that are small enough to tackle in one session. - -### 2a. Google AI model and dimensions outdated - -Docs say `text-embedding-004` at 768d. Codebase default is `gemini-embedding-2-preview` at 3072d. - -**`docs/architecture/embeddings.mdx`** - -- Line 14: Provider comparison table — update Google AI row: Dimensions `768` → `3072`, Model `text-embedding-004` → `gemini-embedding-2-preview` -- Line 40–50: Google AI setup code block — change model env var default to `gemini-embedding-2-preview`, update schema reference to reflect 3072d -- "Choose Google AI if" recommendation — update dimension mention (768 → 3072) - -**`docs/getting-started/configuration.mdx`** - -- Line 33: Table row — change `GOOGLE_EMBEDDING_MODEL` default from `text-embedding-004` → `gemini-embedding-2-preview` -- Lines 50–53: Google AI config block — update model name and dimensions (768 → 3072) - -### 2b. Missing `health_check` MCP tool - -**`docs/mcp-tools/index.mdx`** - -- Line 1: Change "25 MCP tools" → "26 MCP tools" in title/description -- Add `health_check` entry under a new "Diagnostics" section (or append to "Stats" section) with short description and link to new page - -**Create `docs/mcp-tools/health-check.mdx`** - -- Title: `health_check` -- Description: Per-component diagnostic check (database, documents, chunks, memory, conversations, insights). Use as the first step when diagnosing infrastructure issues. -- Parameters: none (or optional `components` filter if supported) -- Example response: show healthy and degraded component states -- Error handling: standard tool error format -- Related tools: `get_stats` - ---- - -## Phase 3 — Missing CLI commands + model ID typo - -### 3a. Missing CLI converters in docs - -**`docs/cli/index.mdx`** - -- Unified Converter commands table (around line 47): Add four missing rows: - - `spotify ` — Convert Spotify data export - - `reddit ` — Convert Reddit data export - - `facebook ` — Convert Facebook data export - - `instagram ` — Convert Instagram data export -- Available Commands section: Add `scan` and `split` entries with links to their new pages -- Next Steps links: Add links to new `scan.mdx` and `split.mdx` - -**Create `docs/cli/scan.mdx`** - -- Title: Scan — Analyze files before upload -- Command: `pnpm scan -- [options]` -- What it does: inspects converted markdown files to report file sizes, estimated chunk counts, and heading structure; flags oversized files -- Options table: `--all`, `--max-file-size `, `--max-chunks ` -- Example output (tabular: file, size, estimated chunks, status) -- Use case: run before `pnpm upload` to identify files needing splitting -- Next steps: link to `split.mdx` and `batch-upload.mdx` - -**Create `docs/cli/split.mdx`** - -- Title: Split — Break large markdown files into chunks -- Command: `pnpm split -- [options]` -- What it does: splits large markdown files at heading boundaries; preserves frontmatter; adds linking metadata -- Options table: `--split-level `, `--target-chunks `, `--output `, `--only-oversized`, `--dry-run`, `-r` (recursive) -- Example: a 2000-line file split into 4 files at `##` headings -- Next steps: link to `batch-upload.mdx` - -### 3b. Model ID typo - -**`docs/getting-started/configuration.mdx`** - -- Line 142: Change `EXTRACTION_MODEL=claude-haiku-4-5-20250501` → `EXTRACTION_MODEL=claude-haiku-4-5-20251001` - ---- - -## Phase 4 — A2A protocol docs + residual cleanup - -### 4a. A2A protocol documentation - -README lists "Agent Discovery — A2A protocol at `/.well-known/agent.json`" as a feature with no corresponding docs. - -**Create `docs/architecture/a2a-protocol.mdx`** - -- Title: Agent-to-Agent (A2A) Protocol -- What it is: standard discovery mechanism allowing other AI agents to find and interact with this knowledge server -- Endpoint: `GET /.well-known/agent.json` — returns agent card (name, description, capabilities, MCP endpoint) -- How to use: point an orchestrator agent at the server URL; it auto-discovers available tools -- Example agent card JSON -- Security note: endpoint is publicly readable by design (no auth required), but actual tool calls still require `API_BEARER_TOKEN` -- Related: link to MCP Tools overview - -**`docs/architecture/index.mdx`** - -- Transport layer section: add A2A as a fourth transport alongside MCP, REST, WebSocket; link to new page - -**`docs/getting-started/introduction.mdx`** - -- Key features list: add "Agent Discovery" feature entry matching README wording - -### 4b. Residual cleanup - -**`docs/mcp-tools/index.mdx`** (if not caught in Phase 1) - -- Confirm pg-analyze description no longer mentions "Supabase client" after Phase 1 edit -- Confirm error table "Database not configured" row uses `DATABASE_URL` after Phase 1 edit +The docs-sync backlog (Phases 1–4) is complete — those documentation fixes shipped and are recorded in `CHANGELOG.md` under `[0.4.0]`. What remains is the server-side Memory System Roadmap below. --- diff --git a/docs/ENV.md b/docs/ENV.md index 6d1a4ef..7a8c400 100644 --- a/docs/ENV.md +++ b/docs/ENV.md @@ -15,8 +15,8 @@ This file centralizes runtime configuration for Textrawl server, MCP tools, CLI, | `DATABASE_URL` | Yes (DB features) | Neon (or any PostgreSQL) pooled connection string | | `EMBEDDING_PROVIDER` | No | `openai` (default), `ollama`, or `google` | | `OPENAI_API_KEY` | Required for OpenAI | Embedding API key (`text-embedding-3-small`, 1536d) | -| `GOOGLE_AI_API_KEY` | Required for Google | Google AI API key (`text-embedding-004`, 768d) | -| `GOOGLE_EMBEDDING_MODEL` | No | Google embedding model (default: `text-embedding-004`) | +| `GOOGLE_AI_API_KEY` | Required for Google | Google AI API key (`gemini-embedding-2-preview`, 3072d) | +| `GOOGLE_EMBEDDING_MODEL` | No | Google embedding model (default: `gemini-embedding-2-preview`) | | `OLLAMA_BASE_URL` | Required for Ollama | Local/remote Ollama base URL | | `OLLAMA_MODEL` | Required for Ollama | Embedding model (e.g. `nomic-embed-text`) | | `API_BEARER_TOKEN` | Strongly recommended; required in prod unless OAuth | Bearer auth for `/mcp` and `/api/upload` | @@ -40,7 +40,7 @@ This file centralizes runtime configuration for Textrawl server, MCP tools, CLI, | Variable | Required | Notes | |---|---|---| | `ANTHROPIC_API_KEY` | If extraction enabled | Key for `extract_memories` | -| `EXTRACTION_MODEL` | No | Default: `claude-haiku-4-5-20250501` | +| `EXTRACTION_MODEL` | No | Default: `claude-haiku-4-5-20251001` | | `INSIGHT_MODEL` | No | Default: `claude-sonnet-4-6-20250514` | ## Optional / advanced diff --git a/docs/architecture/a2a-protocol.mdx b/docs/architecture/a2a-protocol.mdx new file mode 100644 index 0000000..6279c49 --- /dev/null +++ b/docs/architecture/a2a-protocol.mdx @@ -0,0 +1,91 @@ +--- +title: Agent-to-Agent (A2A) Protocol +description: Agent discovery and task delegation via the A2A protocol +--- + +textrawl implements the **Agent-to-Agent (A2A) protocol**, a standard discovery mechanism that lets other AI agents find this knowledge server, learn its capabilities, and delegate natural-language tasks to it. This makes Textrawl usable as a knowledge backend inside a larger multi-agent system, not just as an MCP server for a single client. + +## Agent Card + +```text +GET /.well-known/agent.json +``` + +A public, unauthenticated endpoint returning the agent card — a JSON description of the agent's identity, capabilities, and skills. An orchestrator agent points at the server URL and reads this card to auto-discover what Textrawl can do. + +### Example agent card + +```json +{ + "name": "Textrawl", + "description": "Personal knowledge base agent — search, store, and discover connections across your documents, memories, and conversations.", + "url": "https://your-server.example.com", + "version": "0.2.0", + "capabilities": { + "streaming": false, + "pushNotifications": false + }, + "skills": [ + { "id": "search", "name": "Search Knowledge", "description": "Search across all personal knowledge including documents, memories, conversations, and insights." }, + { "id": "save", "name": "Save Knowledge", "description": "Save notes, URLs, or documents to the knowledge base." }, + { "id": "memory", "name": "Memory Graph", "description": "Store and query facts about entities — people, projects, concepts, and their relationships." }, + { "id": "insights", "name": "Discover Insights", "description": "Find connections and patterns across your knowledge base." } + ], + "defaultInputModes": ["text"], + "defaultOutputModes": ["text"], + "authentication": { "schemes": ["bearer"] } +} +``` + +The `url` field is derived from `OAUTH_SERVER_URL` (falling back to `http://localhost:`), so set that env var in production to advertise the correct origin. + +## Task Delegation + +```text +POST /.well-known/agent/tasks +``` + +Accepts an A2A task message and routes its text through Textrawl's unified search pipeline (documents + memories + conversations, gated by `ENABLE_MEMORY` / `ENABLE_CONVERSATIONS`). Unlike the agent card, this endpoint **requires authentication** — send `Authorization: Bearer `. + +### Request + +```json +{ + "message": { + "parts": [{ "type": "text", "text": "What did I learn about pgvector indexing?" }] + } +} +``` + +### Response + +```json +{ + "id": "task-1718240000000", + "status": { "state": "completed" }, + "output": { + "parts": [{ "type": "text", "text": "[1] [document] pgvector notes: HNSW indexes trade build time for query speed..." }] + }, + "metadata": { + "resultCount": 3, + "counts": { "documents": 2, "memories": 1, "conversations": 0 }, + "sources": [{ "type": "document", "id": "550e8400-...", "title": "pgvector notes" }] + } +} +``` + +## Security + +- `GET /.well-known/agent.json` is **publicly readable by design** — the agent card exposes only capability metadata, no knowledge content. +- `POST /.well-known/agent/tasks` is protected by bearer auth. Discovery is open; actually querying your knowledge still requires `API_BEARER_TOKEN`. + +## How to Use + +1. Deploy Textrawl with `OAUTH_SERVER_URL` (or a reverse-proxy origin) and `API_BEARER_TOKEN` set. +2. Point an orchestrator agent at the server origin; it fetches `/.well-known/agent.json` and registers the four skills. +3. The orchestrator delegates natural-language tasks via `POST /.well-known/agent/tasks` with the bearer token. + +## Related + +- [MCP Tools Overview](/docs/mcp-tools) - The full tool surface available over MCP +- [Architecture Overview](/docs/architecture) - Transports, request flow, and components diff --git a/docs/architecture/embeddings.mdx b/docs/architecture/embeddings.mdx index a37ee0f..c8b0ec8 100644 --- a/docs/architecture/embeddings.mdx +++ b/docs/architecture/embeddings.mdx @@ -11,8 +11,8 @@ textrawl supports three embedding providers: OpenAI (cloud), Google AI (cloud), |---------|--------|-----------|--------| | Location | Cloud | Cloud | Local | | Privacy | Data sent to API | Data sent to API | Data stays local | -| Dimensions | 1536 | 768 | 1024 or 768 | -| Model | text-embedding-3-small | text-embedding-004 | nomic-embed-text / v2-moe | +| Dimensions | 1536 | 3072 | 1024 or 768 | +| Model | text-embedding-3-small | gemini-embedding-2-preview | nomic-embed-text / v2-moe | | Cost | Pay per token | Free tier available | Free | | Speed | ~100ms/request | ~80ms/request | ~50ms/request | | Setup | API key | API key | Docker container | @@ -38,10 +38,10 @@ psql "$DATABASE_URL" -f scripts/setup-db.sql # .env EMBEDDING_PROVIDER=google GOOGLE_AI_API_KEY=your-google-ai-api-key -GOOGLE_EMBEDDING_MODEL=text-embedding-004 +GOOGLE_EMBEDDING_MODEL=gemini-embedding-2-preview ``` -Use the database schema for 768 dimensions: +Use the database schema for 3072 dimensions: ```bash # Apply via psql (or any Postgres SQL editor) @@ -87,7 +87,7 @@ To switch: ## Quality Comparison -The table below compares published MTEB benchmarks for OpenAI and Ollama's nomic-embed-text. Google AI's text-embedding-004 performs comparably on standard retrieval benchmarks but is not included due to differing evaluation methodology. +The table below compares published MTEB benchmarks for OpenAI and Ollama's nomic-embed-text. Google AI's gemini-embedding-2-preview performs comparably on standard retrieval benchmarks but is not included due to differing evaluation methodology. | Benchmark | OpenAI | nomic-embed-text | |-----------|--------|------------------| @@ -108,7 +108,7 @@ OpenAI has a slight edge in benchmarks, but Ollama offers privacy and no API cos **Choose Google AI if:** - You want cloud convenience with a free tier -- 768-dimension embeddings are sufficient +- High-dimensional (3072d) embeddings suit your retrieval quality needs - You already use Google Cloud **Choose Ollama if:** diff --git a/docs/architecture/index.mdx b/docs/architecture/index.mdx index fa54713..e570648 100644 --- a/docs/architecture/index.mdx +++ b/docs/architecture/index.mdx @@ -10,7 +10,7 @@ This section covers the technical details of how textrawl works under the hood. textrawl is built on a modern stack optimized for semantic search: - **Runtime:** Node.js 22+ with ES modules -- **Database:** Supabase PostgreSQL with pgvector extension +- **Database:** Neon PostgreSQL with pgvector extension (any Postgres with pgvector works) - **Transport:** HTTP (REST API, MCP over StreamableHTTP, WebSocket events) - **Embeddings:** OpenAI, Google AI, or Ollama (configurable) @@ -37,7 +37,7 @@ textrawl is built on a modern stack optimized for semantic search: ┌─────┴─────┐ ▼ ▼ ┌─────────┐ ┌─────────┐ - │ Supabase│ │ OpenAI/ │ + │ Postgres│ │ OpenAI/ │ │ DB │ │ Ollama │ └─────────┘ └─────────┘ ``` @@ -67,7 +67,7 @@ Vector representations that capture semantic meaning, enabling similarity search | Provider | Model | Dimensions | |----------|-------|------------| | OpenAI | text-embedding-3-small | 1536 | -| Google AI | text-embedding-004 | 768 | +| Google AI | gemini-embedding-2-preview | 3072 | | Ollama | nomic-embed-text | 1024 | | Ollama | nomic-embed-text-v2-moe | 768 | @@ -100,7 +100,7 @@ Vector representations that capture semantic meaning, enabling similarity search ## Transport Layer -textrawl exposes three transport interfaces: MCP (for AI assistants), REST API (for programmatic access), and WebSocket (for real-time events). +textrawl exposes four transport interfaces: MCP (for AI assistants), REST API (for programmatic access), WebSocket (for real-time events), and the A2A protocol (for agent-to-agent discovery). ### MCP @@ -122,6 +122,12 @@ Standard HTTP endpoints for direct integration: Real-time event streaming for document ingestion, insight discovery, and other background operations. +### A2A Protocol + +Agent-to-agent discovery via a public agent card at `GET /.well-known/agent.json`, letting orchestrator agents find the server and its capabilities. + +[Learn about the A2A protocol →](/docs/architecture/a2a-protocol) + ## Rate Limiting | Endpoint | Limit | diff --git a/docs/architecture/meta.json b/docs/architecture/meta.json index a29fb5c..77730f3 100644 --- a/docs/architecture/meta.json +++ b/docs/architecture/meta.json @@ -1,4 +1,4 @@ { "title": "Architecture", - "pages": ["index", "hybrid-search", "chunking", "embeddings"] + "pages": ["index", "hybrid-search", "chunking", "embeddings", "a2a-protocol"] } diff --git a/docs/plans/2026-05-31-large-upload-gcs-resumable-plan.md b/docs/archive/2026-05-31-large-upload-gcs-resumable-plan.md similarity index 100% rename from docs/plans/2026-05-31-large-upload-gcs-resumable-plan.md rename to docs/archive/2026-05-31-large-upload-gcs-resumable-plan.md diff --git a/docs/plans/2026-06-01-phase4-cloud-tasks-impl.md b/docs/archive/2026-06-01-phase4-cloud-tasks-impl.md similarity index 100% rename from docs/plans/2026-06-01-phase4-cloud-tasks-impl.md rename to docs/archive/2026-06-01-phase4-cloud-tasks-impl.md diff --git a/docs/plans/2026-06-06-phase4-cloud-run-activation.md b/docs/archive/2026-06-06-phase4-cloud-run-activation.md similarity index 100% rename from docs/plans/2026-06-06-phase4-cloud-run-activation.md rename to docs/archive/2026-06-06-phase4-cloud-run-activation.md diff --git a/docs/plans/2026-06-10-phase5-handler-registry-zip-impl.md b/docs/archive/2026-06-10-phase5-handler-registry-zip-impl.md similarity index 100% rename from docs/plans/2026-06-10-phase5-handler-registry-zip-impl.md rename to docs/archive/2026-06-10-phase5-handler-registry-zip-impl.md diff --git a/docs/plans/2026-06-10-phase6-dashboard-upload-ux-impl.md b/docs/archive/2026-06-10-phase6-dashboard-upload-ux-impl.md similarity index 100% rename from docs/plans/2026-06-10-phase6-dashboard-upload-ux-impl.md rename to docs/archive/2026-06-10-phase6-dashboard-upload-ux-impl.md diff --git a/docs/IMPROVEMENT_TRACKER.md b/docs/archive/IMPROVEMENT_TRACKER.md similarity index 100% rename from docs/IMPROVEMENT_TRACKER.md rename to docs/archive/IMPROVEMENT_TRACKER.md diff --git a/docs/cli/batch-upload.mdx b/docs/cli/batch-upload.mdx index 1bcbd40..1c334fb 100644 --- a/docs/cli/batch-upload.mdx +++ b/docs/cli/batch-upload.mdx @@ -1,9 +1,9 @@ --- title: Batch Upload -description: Upload converted files to Supabase with automatic embedding +description: Upload converted files to your knowledge base with automatic embedding --- -Upload converted markdown files to your Supabase knowledge base with automatic chunking and embedding. +Upload converted markdown files to your knowledge base with automatic chunking and embedding. ## Usage @@ -40,7 +40,7 @@ pnpm upload -- ./converted/ --dry-run 3. **Parse Frontmatter**: Extract metadata 4. **Chunk Content**: Split into ~512 token chunks 5. **Generate Embeddings**: Via OpenAI or Ollama -6. **Store in Supabase**: Documents + chunks + embeddings +6. **Store in Postgres**: Documents + chunks + embeddings 7. **Update Manifest**: Track uploaded files ## Manifest File @@ -67,7 +67,7 @@ Use `--force` to ignore the manifest and re-upload everything. Uploads are rate-limited to avoid API throttling: - **Embedding API**: 60 requests/minute -- **Supabase**: 100 requests/minute +- **Server API**: 100 requests/minute ## Error Handling diff --git a/docs/cli/index.mdx b/docs/cli/index.mdx index 54cb743..c83055a 100644 --- a/docs/cli/index.mdx +++ b/docs/cli/index.mdx @@ -30,6 +30,18 @@ Convert HTML files and web pages to searchable markdown. [HTML conversion →](/docs/cli/html-conversion) +### scan + +Analyze converted markdown files for upload readiness and flag oversized files. + +[Scan →](/docs/cli/scan) + +### split + +Break large markdown files into uploadable parts at heading boundaries. + +[Split →](/docs/cli/split) + ### upload Upload converted files to your knowledge base with automatic embedding. @@ -50,6 +62,10 @@ pnpm convert -- [options] | `eml ` | Convert EML file(s) or directory | | `html ` | Convert HTML file(s) or directory | | `takeout ` | Convert Google Takeout archive | +| `spotify ` | Convert Spotify data export | +| `reddit ` | Convert Reddit data export | +| `facebook ` | Convert Facebook data export | +| `instagram ` | Convert Instagram data export | | `auto ` | Auto-detect format and convert | ## Common Options @@ -135,4 +151,6 @@ pnpm ui - [MBOX Conversion](/docs/cli/mbox-conversion) - Import email archives - [HTML Conversion](/docs/cli/html-conversion) - Convert web pages +- [Scan](/docs/cli/scan) - Check files for upload readiness +- [Split](/docs/cli/split) - Break large files into parts - [Batch Upload](/docs/cli/batch-upload) - Upload to knowledge base diff --git a/docs/cli/meta.json b/docs/cli/meta.json index b85dcdd..0592a4a 100644 --- a/docs/cli/meta.json +++ b/docs/cli/meta.json @@ -1,4 +1,4 @@ { "title": "CLI Tools", - "pages": ["index", "mbox-conversion", "html-conversion", "batch-upload"] + "pages": ["index", "mbox-conversion", "html-conversion", "scan", "split", "batch-upload"] } diff --git a/docs/cli/scan.mdx b/docs/cli/scan.mdx new file mode 100644 index 0000000..6b1fe19 --- /dev/null +++ b/docs/cli/scan.mdx @@ -0,0 +1,66 @@ +--- +title: Scan +description: Analyze converted markdown files for upload readiness +--- + +`scan` inspects converted markdown files before upload. It reports file sizes, estimated chunk counts, and heading structure, and flags files that exceed the upload limits so you can split them first. + +It is **read-only** — it never modifies or uploads anything. + +## Usage + +```bash +pnpm scan -- [options] +``` + +```bash +# Scan a directory, showing only problematic files +pnpm scan -- ./converted/ + +# Show every file, not just oversized ones +pnpm scan -- ./converted/ --all + +# Detailed analysis of a single file (heading tree + split suggestions) +pnpm scan -- ./converted/large-file.md +``` + +## Options + +| Option | Default | Description | +|--------|---------|-------------| +| `-r, --recursive` | `true` | Scan subdirectories | +| `--pattern ` | `**/*.md` | Glob pattern for files to scan | +| `--max-file-size ` | `20` | File size threshold (MB) before a file is flagged | +| `--max-chunks ` | `500` | Estimated chunk-count threshold before a file is flagged | +| `--all` | `false` | Show all files, not just ones that exceed limits | +| `--format ` | `table` | Output format: `table` or `json` | + +## Example Output + +```text +=== Upload Readiness Scan === +File size limit: 20 MB | Chunk limit: 500 + + STATUS SIZE CHUNKS FILE + ✓ OK 1.2 MB ~180 emails/2024-01.md + ✗ CHUNKS 4.8 MB ~720 archive/all-threads.md + +=== Summary === +Total: 2 | Uploadable: 1 | Need splitting: 1 +Estimated total chunks: ~900 + +=== Suggested Splits === +archive/all-threads.md: 6 h2 headings -> split at h2 would create ~7 parts + Run: pnpm split -- archive/all-threads.md +``` + +A file is flagged when it exceeds either `--max-file-size` (`✗ SIZE`) or `--max-chunks` (`✗ CHUNKS`). For a single file, the scan additionally prints the full heading tree and suggested split points. + +## Use Case + +Run `scan` after `pnpm convert` and before `pnpm upload` to find files that would fail the upload chunk limit. For any flagged file, run [`split`](/docs/cli/split) to break it into uploadable parts. + +## Next Steps + +- [Split](/docs/cli/split) - Break large files into uploadable parts +- [Batch Upload](/docs/cli/batch-upload) - Upload converted files diff --git a/docs/cli/split.mdx b/docs/cli/split.mdx new file mode 100644 index 0000000..b09370f --- /dev/null +++ b/docs/cli/split.mdx @@ -0,0 +1,53 @@ +--- +title: Split +description: Break large markdown files into uploadable parts +--- + +`split` breaks large markdown files into smaller parts at heading boundaries. It preserves YAML frontmatter on every part and adds linking metadata so the parts stay associated. Use it on files that [`scan`](/docs/cli/scan) flags as oversized. + +## Usage + +```bash +pnpm split -- [options] +``` + +```bash +# Split a single file at h2 headings (default) +pnpm split -- ./converted/large-file.md + +# Preview the split without writing files +pnpm split -- ./converted/large-file.md --dry-run + +# Split only files that exceed upload limits, recursively +pnpm split -- ./converted/ -r --only-oversized +``` + +## Options + +| Option | Default | Description | +|--------|---------|-------------| +| `--split-level ` | `2` | Heading level (1–6) to split at | +| `--target-size ` | `15` | Target maximum size per part (MB) | +| `--target-chunks ` | `400` | Target maximum estimated chunks per part | +| `--suffix ` | `-part-{n}` | Filename suffix for parts (`{n}` = part number) | +| `--only-oversized` | `false` | Only split files that exceed the upload limits below | +| `--max-file-size ` | `20` | Oversized file-size threshold (with `--only-oversized`) | +| `--max-chunks ` | `500` | Oversized chunk-count threshold (with `--only-oversized`) | +| `--pattern ` | `**/*.md` | Glob pattern when given a directory | +| `-r, --recursive` | `false` | Process directories recursively | +| `-o, --output ` | `source dir` | Output directory for split parts (defaults to the source file's directory) | +| `--dry-run` | `false` | Preview the split plan without writing files | +| `-v, --verbose` | `false` | Verbose logging | + +## How It Works + +1. The file is split at the chosen heading level (default `##`/h2). If a single section still exceeds the target size or chunk count, it is further split at paragraph boundaries. +2. Each part keeps the original frontmatter, with added linking metadata so the parts remain grouped. +3. Parts are named with the `--suffix` pattern — e.g. `notes-part-1.md`, `notes-part-2.md`. + +A 2000-line file with four `##` sections splits into roughly four parts, each small enough to upload. + +## Next Steps + +- [Scan](/docs/cli/scan) - Identify which files need splitting +- [Batch Upload](/docs/cli/batch-upload) - Upload the split parts diff --git a/docs/concepts/index.mdx b/docs/concepts/index.mdx index 07305b3..cf8928a 100644 --- a/docs/concepts/index.mdx +++ b/docs/concepts/index.mdx @@ -22,7 +22,7 @@ textrawl combines two search strategies for best results: Vector embeddings are numerical representations of text that capture semantic meaning: - OpenAI's `text-embedding-3-small` (1536 dimensions) -- Google AI's `text-embedding-004` (768 dimensions) +- Google AI's `gemini-embedding-2-preview` (3072 dimensions) - Ollama's `nomic-embed-text` (1024 dimensions) or `nomic-embed-text-v2-moe` (768 dimensions) - Stored in PostgreSQL with pgvector @@ -65,7 +65,7 @@ Documents (PDF, MBOX, HTML, MD) └────────┬─────────┘ ▼ ┌──────────┐ - │ Supabase │ + │ Postgres │ │ (pgvector)│ └──────────┘ ``` diff --git a/docs/getting-started/configuration.mdx b/docs/getting-started/configuration.mdx index 9e5101a..33dd497 100644 --- a/docs/getting-started/configuration.mdx +++ b/docs/getting-started/configuration.mdx @@ -30,7 +30,7 @@ Choose between OpenAI (cloud), Google AI (cloud), or Ollama (local): | `EMBEDDING_PROVIDER` | No | `openai` (default), `ollama`, or `google` | | `OPENAI_API_KEY` | If OpenAI | Your OpenAI API key | | `GOOGLE_AI_API_KEY` | If Google | Your Google AI API key | -| `GOOGLE_EMBEDDING_MODEL` | If Google | Model name (default: `text-embedding-004`) | +| `GOOGLE_EMBEDDING_MODEL` | If Google | Model name (default: `gemini-embedding-2-preview`) | | `OLLAMA_BASE_URL` | If Ollama | Ollama server URL | | `OLLAMA_MODEL` | If Ollama | Model name (default: `nomic-embed-text`) | @@ -50,11 +50,11 @@ OPENAI_API_KEY=sk-proj-... ```bash EMBEDDING_PROVIDER=google GOOGLE_AI_API_KEY=your-google-ai-api-key -GOOGLE_EMBEDDING_MODEL=text-embedding-004 +GOOGLE_EMBEDDING_MODEL=gemini-embedding-2-preview ``` -- Model: `text-embedding-004` -- Dimensions: 768 +- Model: `gemini-embedding-2-preview` +- Dimensions: 3072 - Schema: `scripts/setup-db-google.sql` **Ollama Configuration:** @@ -137,12 +137,12 @@ Required only when `ENABLE_MEMORY_EXTRACTION=true`: | Variable | Required | Default | Description | |----------|----------|---------|-------------| | `ANTHROPIC_API_KEY` | If extraction | - | Anthropic API key for Claude | -| `EXTRACTION_MODEL` | No | `claude-haiku-4-5-20250501` | Model for entity extraction | +| `EXTRACTION_MODEL` | No | `claude-haiku-4-5-20251001` | Model for entity extraction | ```bash ENABLE_MEMORY_EXTRACTION=true ANTHROPIC_API_KEY=sk-ant-... -EXTRACTION_MODEL=claude-haiku-4-5-20250501 +EXTRACTION_MODEL=claude-haiku-4-5-20251001 ``` ### Insight Configuration diff --git a/docs/getting-started/introduction.mdx b/docs/getting-started/introduction.mdx index 116a187..c9d26f7 100644 --- a/docs/getting-started/introduction.mdx +++ b/docs/getting-started/introduction.mdx @@ -83,7 +83,7 @@ Access your knowledge base beyond MCP clients: - **REST Endpoints**: Search, list documents, upload files over HTTP - **WebSocket Events**: Real-time notifications for ingestion, extraction, and insight discovery -- **Agent Discovery**: A2A protocol at `/.well-known/agent.json` for agent-to-agent interaction +- **Agent Discovery**: [A2A protocol](/docs/architecture/a2a-protocol) at `/.well-known/agent.json` for agent-to-agent interaction ### Self-Hosted diff --git a/docs/guides/email-import.mdx b/docs/guides/email-import.mdx index 2b2c400..1fa9441 100644 --- a/docs/guides/email-import.mdx +++ b/docs/guides/email-import.mdx @@ -46,6 +46,6 @@ pnpm convert -- mbox personal.mbox -t personal ## Privacy Considerations -- Emails are stored in your Supabase instance +- Emails are stored in your Postgres database - Embeddings go through OpenAI/Ollama - Consider Ollama for sensitive data diff --git a/docs/guides/large-upload-e2e-preflight.md b/docs/guides/large-upload-e2e-preflight.md index b1103e5..e60699c 100644 --- a/docs/guides/large-upload-e2e-preflight.md +++ b/docs/guides/large-upload-e2e-preflight.md @@ -25,8 +25,8 @@ here): `lifecycle.json`, `README.md`) - Cloud Tasks queue + OIDC invoker SA + IAM → [`infra/cloud-tasks/`](../../infra/cloud-tasks/) (`setup.sh`, `README.md`) -- Cloud Run env activation → [`docs/plans/2026-06-06-phase4-cloud-run-activation.md`](../plans/2026-06-06-phase4-cloud-run-activation.md) -- API contract + smoke checklist → [`docs/plans/2026-05-31-large-upload-gcs-resumable-plan.md`](../plans/2026-05-31-large-upload-gcs-resumable-plan.md) §4, §13 +- Cloud Run env activation → [`docs/archive/2026-06-06-phase4-cloud-run-activation.md`](../archive/2026-06-06-phase4-cloud-run-activation.md) +- API contract + smoke checklist → [`docs/archive/2026-05-31-large-upload-gcs-resumable-plan.md`](../archive/2026-05-31-large-upload-gcs-resumable-plan.md) §4, §13 - Deployment reference → [`docs/guides/cloud-run-deployment.mdx`](./cloud-run-deployment.mdx) --- diff --git a/docs/mcp-tools/health-check.mdx b/docs/mcp-tools/health-check.mdx new file mode 100644 index 0000000..bffd236 --- /dev/null +++ b/docs/mcp-tools/health-check.mdx @@ -0,0 +1,99 @@ +--- +title: health_check +description: Per-component diagnostic check across all textrawl subsystems +--- + +Quick diagnostic check across all textrawl subsystems. Returns a pass/fail result for each component with error details on failure. Use this as the **first call** when other tools are misbehaving — it isolates whether the problem is the database connection, a missing schema, or a specific feature. + +This tool is **read-only** and is always registered (independent of the `ENABLE_*` feature flags). + +## Parameters + +None. Call with an empty object: + +```json +{} +``` + +## Components Checked + +| Component | What it verifies | +|-----------|------------------| +| `database` | A `SELECT 1` round-trip succeeds; reports `latencyMs` | +| `documents` | The `documents` table is readable; reports row `count` | +| `chunks` | The `chunks` table is readable; reports row `count` | +| `memory` | The `memory_entities` table is readable; reports `entities` count | +| `conversations` | The `conversation_sessions` table is readable; reports `sessions` count | +| `insights` | The `proactive_insights` table is readable; reports `count` | + +## Response + +```json +{ + "status": "healthy", + "checks": { + "database": { "ok": true, "latencyMs": 12 }, + "documents": { "ok": true, "count": 150 }, + "chunks": { "ok": true, "count": 1840 }, + "memory": { "ok": true, "entities": 25 }, + "conversations": { "ok": true, "sessions": 15 }, + "insights": { "ok": true, "count": 8 } + }, + "timestamp": "2026-06-13T01:00:00.000Z" +} +``` + +### Degraded example + +When some components fail (for example, an optional schema was never installed), the failing checks include an `error` and the overall `status` becomes `degraded`: + +```json +{ + "status": "degraded", + "checks": { + "database": { "ok": true, "latencyMs": 14 }, + "documents": { "ok": true, "count": 150 }, + "chunks": { "ok": true, "count": 1840 }, + "memory": { "ok": false, "error": "relation \"memory_entities\" does not exist" }, + "conversations": { "ok": false, "error": "relation \"conversation_sessions\" does not exist" }, + "insights": { "ok": false, "error": "relation \"proactive_insights\" does not exist" } + }, + "timestamp": "2026-06-13T01:00:00.000Z" +} +``` + +## Output Schema + +This tool returns `structuredContent` alongside the text response, using the same canonical keys regardless of the `COMPACT_RESPONSES` setting. + +| Field | Type | Description | +|-------|------|-------------| +| `status` | enum | Overall status: `healthy` (all pass), `degraded` (some pass), `unhealthy` (none pass) | +| `checks` | object | Map of component name → check result | +| `checks..ok` | boolean | Whether the component check passed | +| `checks..latencyMs` | integer? | Database round-trip latency (database check only) | +| `checks..count` | integer? | Row count (documents, chunks, insights) | +| `checks..entities` | integer? | Entity count (memory check) | +| `checks..sessions` | integer? | Session count (conversations check) | +| `checks..error` | string? | Error message when `ok` is false | +| `timestamp` | string | ISO 8601 timestamp of the check | + +## Status Logic + +| Status | Condition | +|--------|-----------| +| `healthy` | Every component check passed | +| `degraded` | At least one passed and at least one failed | +| `unhealthy` | Every component check failed | + +## Errors + +| Error | Cause | Fix | +|-------|-------|-----| +| Database not configured | Missing database connection | Set `DATABASE_URL` | + +A missing optional schema (memory, conversations, insights) does **not** raise an error — it surfaces as a failed component check in the response so partial outages remain visible. + +## Related Tools + +- [`get_stats`](/docs/mcp-tools/get-stats) - Detailed per-feature statistics once the system is healthy diff --git a/docs/mcp-tools/index.mdx b/docs/mcp-tools/index.mdx index 876cf47..9f7a38c 100644 --- a/docs/mcp-tools/index.mdx +++ b/docs/mcp-tools/index.mdx @@ -1,9 +1,9 @@ --- title: MCP Tools Overview -description: Complete reference for textrawl's 25 MCP tools +description: Complete reference for textrawl's 26 MCP tools --- -textrawl exposes 25 tools via the Model Context Protocol (MCP). These tools allow Claude and other MCP clients to search, retrieve, create, and manage documents, memories, conversations, and insights in your knowledge base. +textrawl exposes 26 tools via the Model Context Protocol (MCP). These tools allow Claude and other MCP clients to search, retrieve, create, and manage documents, memories, conversations, and insights in your knowledge base. ## Document Tools @@ -121,7 +121,7 @@ Dismiss an insight so it no longer appears in results. [View documentation →](/docs/mcp-tools/dismiss-insight) -## Stats +## Stats & Diagnostics ### get_stats @@ -129,6 +129,12 @@ Get statistics across all features with scope selection: `all`, `knowledge`, `me [View documentation →](/docs/mcp-tools/get-stats) +### health_check + +Per-component diagnostic check (database, documents, chunks, memory, conversations, insights). Use as the first step when diagnosing infrastructure issues. + +[View documentation →](/docs/mcp-tools/health-check) + ## Unified Tools ### ask diff --git a/docs/mcp-tools/meta.json b/docs/mcp-tools/meta.json index cc8f754..635a698 100644 --- a/docs/mcp-tools/meta.json +++ b/docs/mcp-tools/meta.json @@ -23,8 +23,9 @@ "get-insights", "discover-connections", "dismiss-insight", - "---Stats---", + "---Stats & Diagnostics---", "get-stats", + "health-check", "---Unified Tools---", "ask", "daily-briefing", diff --git a/docs/mcp-tools/pg-analyze.mdx b/docs/mcp-tools/pg-analyze.mdx index e259a13..fb1ba7b 100644 --- a/docs/mcp-tools/pg-analyze.mdx +++ b/docs/mcp-tools/pg-analyze.mdx @@ -96,7 +96,7 @@ When `save` is `true`, the report is persisted to `PG_REPORT_DIR` (default `./re ## Notes - This tool is **read-only** and **not destructive** — it only reads database statistics and metadata. -- Requires the `DATABASE_URL` environment variable to be set (direct `pg` connection, independent of Supabase). +- Requires the `DATABASE_URL` environment variable to be set (direct `pg` connection, independent of the main database client). - The analysis covers: - **Table stats**: row counts, sizes, growth - **Index health**: unused indexes, duplicate indexes, missing indexes diff --git a/docs/mcp-tools/pg-recommendations.mdx b/docs/mcp-tools/pg-recommendations.mdx index 6976f8a..2668be6 100644 --- a/docs/mcp-tools/pg-recommendations.mdx +++ b/docs/mcp-tools/pg-recommendations.mdx @@ -91,7 +91,7 @@ Each recommendation includes: ## Notes - This tool is **read-only** and **not destructive** — it does not modify your database. -- Requires the `DATABASE_URL` environment variable to be set (direct `pg` connection, independent of Supabase). +- Requires the `DATABASE_URL` environment variable to be set (direct `pg` connection, independent of the main database client). - Use `severity` filtering to focus on the most impactful issues first. - For the full analysis report (table stats, index health, etc.), use [`pg_analyze`](/docs/mcp-tools/pg-analyze) instead. diff --git a/infra/cloud-tasks/README.md b/infra/cloud-tasks/README.md index 292a597..6a00f77 100644 --- a/infra/cloud-tasks/README.md +++ b/infra/cloud-tasks/README.md @@ -1,7 +1,7 @@ # Cloud Tasks provisioning — async upload processing Infra-as-config for the Phase 4 async pipeline -(`docs/plans/2026-06-01-phase4-cloud-tasks-impl.md`). Cloud Tasks delivers an +(`docs/archive/2026-06-01-phase4-cloud-tasks-impl.md`). Cloud Tasks delivers an OIDC-authenticated request to the internal `POST /api/upload/process/:uploadId` endpoint on the existing `textrawl` Cloud Run service; the app verifies the OIDC token itself (the service is public for MCP/API, so this is the access control). @@ -43,7 +43,7 @@ re-run. > **The T4.1–T4.3 code is now merged and deployed.** For the remaining activation > steps — which env vars are still missing, how to resolve the live service URL, > and a copy-paste agent prompt — see -> [docs/plans/2026-06-06-phase4-cloud-run-activation.md](../../docs/plans/2026-06-06-phase4-cloud-run-activation.md). +> [docs/archive/2026-06-06-phase4-cloud-run-activation.md](../../docs/archive/2026-06-06-phase4-cloud-run-activation.md). `setup.sh` prints the exact values. The processing code reads these env vars; they are inert until that code ships, so set them at the same deploy: diff --git a/infra/gcs/README.md b/infra/gcs/README.md index a592330..f406f13 100644 --- a/infra/gcs/README.md +++ b/infra/gcs/README.md @@ -1,7 +1,7 @@ # GCS provisioning — large-upload bucket Infra-as-config for the resumable large-upload workflow -(`docs/plans/2026-05-31-large-upload-gcs-resumable-plan.md`, §3–§7). +(`docs/archive/2026-05-31-large-upload-gcs-resumable-plan.md`, §3–§7). ## Resource diff --git a/llms-full.txt b/llms-full.txt index 37afcf0..74ff276 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -4,7 +4,7 @@ ## Overview -Textrawl is a personal knowledge server with persistent memory, searchable documents, and proactive insights. Access it through the web dashboard, MCP for AI assistants, or the REST API. Backed by Supabase PostgreSQL with vector embeddings. +Textrawl is a personal knowledge server with persistent memory, searchable documents, and proactive insights. Access it through the web dashboard, MCP for AI assistants, or the REST API. Backed by Neon PostgreSQL with vector embeddings. ## Quick Start @@ -480,7 +480,7 @@ Example: ## Postgres Analysis Tools -Enabled when `DATABASE_URL` is configured. Connects directly to Postgres via `pg` (independent of Supabase client). +Enabled when `DATABASE_URL` is configured. Connects directly to Postgres via `pg` (independent of the main database client). ### pg_analyze @@ -506,15 +506,13 @@ Parameters: ## Environment Variables -Required (for Supabase-backed features — not needed for standalone Postgres analysis): -- SUPABASE_URL: Database connection URL -- SUPABASE_SERVICE_KEY: Service role authentication +Required (for database-backed features): +- DATABASE_URL: Postgres connection string (Neon pooled URL recommended; any pgvector-enabled Postgres works). Also powers the pg_analyze tools. Conditional: - OPENAI_API_KEY: For cloud embeddings (text-embedding-3-small, 1536 dims) -- GOOGLE_AI_API_KEY: For Google AI embeddings (text-embedding-004, 768 dims) +- GOOGLE_AI_API_KEY: For Google AI embeddings (gemini-embedding-2-preview, 3072 dims) - OLLAMA_BASE_URL + OLLAMA_MODEL: For local embeddings (nomic-embed-text 1024d, nomic-embed-text-v2-moe 768d) -- DATABASE_URL: Direct Postgres connection for pg_analyze tools (pg_analyze, pg_recommendations, pg_report_history); standalone — does not require Supabase credentials Optional: - PG_REPORT_DIR: Directory for Postgres analysis report files (default ./reports/pg-analysis) @@ -527,7 +525,7 @@ Optional: - ENABLE_INSIGHTS: Enable proactive insight tools (default true) - ENABLE_MEMORY_EXTRACTION: Enable LLM-based memory extraction (default false) - ANTHROPIC_API_KEY: Required for extract_memories tool -- EXTRACTION_MODEL: Model for extraction (default claude-haiku-4-5-20250501) +- EXTRACTION_MODEL: Model for extraction (default claude-haiku-4-5-20251001) - INSIGHT_MODEL: Model for insight synthesis (default claude-sonnet-4-6-20250514) ## CLI Tools @@ -540,7 +538,7 @@ pnpm convert -- html ./saved-pages/ -r pnpm convert -- takeout ./Takeout.zip ``` -### Upload to Supabase +### Upload to Knowledge Base ```bash pnpm upload -- ./converted/ diff --git a/llms.txt b/llms.txt index 75fbb71..a42ddfb 100644 --- a/llms.txt +++ b/llms.txt @@ -2,7 +2,7 @@ > Personal knowledge server — your documents, searchable and AI-ready -Textrawl is a personal knowledge server with persistent memory, searchable documents, and proactive insights. Access it through the web dashboard, MCP for AI assistants, or the REST API. Backed by Supabase PostgreSQL with vector embeddings. +Textrawl is a personal knowledge server with persistent memory, searchable documents, and proactive insights. Access it through the web dashboard, MCP for AI assistants, or the REST API. Backed by Neon PostgreSQL with vector embeddings. ## Server Interfaces @@ -237,7 +237,7 @@ Browse knowledge chronologically within a date range, optionally filtered by top ## Postgres Analysis Tools -Enabled when DATABASE_URL is configured. Connects directly to Postgres (independent of Supabase client). +Enabled when DATABASE_URL is configured. Connects directly to Postgres (independent of the main database client). ### pg_analyze @@ -268,9 +268,9 @@ View past Postgres analysis reports and compare trends. ## Quick Facts -- 25 MCP tools available (5 document/search + 6 memory + 3 conversation + 1 stats + 3 insight + 4 unified + 3 Postgres analysis) -- Embedding: OpenAI (1536 dim), Google AI (768 dim), or Ollama (1024 dim / 768 dim) -- Database: Supabase PostgreSQL with pgvector +- 26 MCP tools available (5 document/search + 6 memory + 3 conversation + 2 stats/diagnostics + 3 insight + 4 unified + 3 Postgres analysis) +- Embedding: OpenAI (1536 dim), Google AI (3072 dim), or Ollama (1024 dim / 768 dim) +- Database: Neon PostgreSQL with pgvector - Chunking: 512 tokens, 50 token overlap, paragraph-aware - Node.js >= 22.0.0 required diff --git a/package.json b/package.json index 9c7acde..424eb7f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "textrawl", - "version": "0.3.0", + "version": "0.4.0", "description": "Personal knowledge server - semantic search, persistent memory, and proactive insights for your documents", "type": "module", "engines": {