jeffgreendesign · jeffgreendesign · Jun 13, 2026 · Jun 13, 2026 · Jun 13, 2026
diff --git a/AGENTS.md b/AGENTS.md
@@ -17,9 +17,9 @@ pnpm inspector
 
 ## Safety rules
 
-- Never log, print, or commit `SUPABASE_SERVICE_KEY` or `API_BEARER_TOKEN`.
-- Keep server-only secrets server-side. Do not place service-role credentials in desktop renderer code, website bundles, or client config files.
-- Treat desktop distribution as client-like: only call server APIs from the desktop app, never embed privileged Supabase keys.
+- Never log, print, or commit `DATABASE_URL` or `API_BEARER_TOKEN`.
+- Keep server-only secrets server-side. Do not place database credentials in desktop renderer code, website bundles, or client config files.
+- Treat desktop distribution as client-like: only call server APIs from the desktop app, never embed privileged database credentials.
 
 ## MCP tool inventory
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,52 @@ All notable changes to Textrawl are documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [0.4.0] - 2026-06-13
+
+### Added
+
+- **Large-upload pipeline**: GCS-backed resumable upload sessions (server initiates the session, the browser PUTs directly to the returned URI) with `/api/upload/init` and `/api/upload/complete`, an idempotent upload state machine, and the `uploads` / `upload_entries` metadata schema (`scripts/setup-db-uploads.sql`)
+- **Cloud Tasks processing**: background processing queue with OIDC authentication and dedupe, plus an internal Cloud Run processing endpoint for async single-file ingestion
+- **Processor handler registry**: pluggable converter registry covering Tier-1 file types (pdf, docx, html, xlsx, csv, json, text) with safe streaming ZIP extraction
+- **Dashboard resumable upload UX**: large-upload flow that resolves at `queued` so processing runs in the background, an honest file-type support matrix, and clear structured upload errors
+- **`scan` and `split` CLI commands**: analyze converted markdown for upload readiness and split oversized files at heading boundaries before upload
+- **Social export converters**: Spotify, Reddit, Facebook, and Instagram data-export converters
+- **Optional Redis-backed rate limiting**: set `REDIS_URL` to share rate-limit counters across server instances (falls back to in-memory)
+- **`health_check` MCP tool**: per-component diagnostic (database, documents, chunks, memory, conversations, insights) — tool count is now 26
+
+### Changed
+
+- **Google AI embedding default** → `gemini-embedding-2-preview` (3072 dimensions). **Breaking**: changing from the previous `text-embedding-004` (768d) requires re-embedding all documents with `scripts/setup-db-google.sql`
+- **Default `EXTRACTION_MODEL`** → `claude-haiku-4-5-20251001`
+- **Provider resilience**: retry transient embedding/Anthropic failures with exponential backoff; Ollama fetch timeout with typed provider errors
+- **Bounded ingestion memory**: cap memory across the upload→process pipeline and defer per-document memory extraction off the processing critical path
+
+### Fixed
+
+- Chunker O(n) force-split to stop OOM on large single-paragraph text
+- Keep OpenAI embedding batches under the 300K-token request cap
+- Tolerate truncated/fenced LLM JSON during memory extraction; raise extraction `max_tokens`
+- Return structured JSON for Multer/413/file-type upload errors; wire `MAX_SINGLE_FILE_SIZE_MB` into the multer limit
+- Dashboard: derive the WebSocket URL from the API base (no localhost fallback in production), route ZIP uploads through the resumable path, and suppress the React #418 hydration warning
+- Move `unzipper` to runtime dependencies so ZIP processing boots on Cloud Run
+- Remove a dead xlsx budget assignment flagged by CodeQL
+
+### Documentation
+
+- Synced docs to code: replaced Supabase setup with Neon / `DATABASE_URL` across installation, configuration, RUNBOOK, troubleshooting, Docker and Cloud Run guides, and per-tool error tables
+- Corrected the Google AI embedding model and dimensions (3072d) across all documentation
+- Added the `health_check` tool reference and updated the tool count from 25 to 26
+- Added `scan` and `split` CLI reference pages and documented the Spotify/Reddit/Facebook/Instagram converters
+- Added an Agent-to-Agent (A2A) protocol reference page
+- Archived completed implementation plans (`docs/plans/`) and the improvement tracker to `docs/archive/`
+
+### Dependencies
+
+- Add `@google-cloud/storage` for GCS resumable uploads
+- Add `@google-cloud/tasks` and `google-auth-library` for Cloud Tasks queueing and OIDC
+- Add `redis` and `rate-limit-redis` for optional shared rate limiting
+- Move `unzipper` to runtime dependencies
+
 ## [0.3.0] - 2026-03-26
 
 ### Added

diff --git a/CLAUDE.md b/CLAUDE.md
@@ -90,7 +90,7 @@ Implementations: `src/tools/*.ts`. Full descriptions and schemas in `README.md`.
 
 ## Postgres analysis tools
 
-- Gated on `DATABASE_URL` env var (direct `pg` connection, independent of Supabase)
+- Gated on `DATABASE_URL` env var (direct `pg` connection, independent of the main database client)
 - MCP tools: `src/tools/pg-analyze.ts`
 - Analysis engine: `src/services/pg-analyze/`
 - CLI: `pnpm pg:analyze` (`scripts/cli/pg-analyze.ts`)

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -8,7 +8,7 @@ Thanks for your interest in contributing! This guide will help you get started.
 
 - Node.js >= 22.0.0
 - Docker (for local database)
-- A Supabase account OR local PostgreSQL with pgvector
+- A Neon account (or any pgvector-enabled Postgres) OR local PostgreSQL with pgvector
 - OpenAI API key OR [Ollama](https://ollama.com) (free, local) for embeddings
 
 ### Quick Start
@@ -24,7 +24,7 @@ pnpm install
 # Run the setup script (generates .env with secure token)
 pnpm setup
 
-# Start local database (optional - or use Supabase)
+# Start local database (optional - or use a hosted Postgres like Neon)
 docker-compose -f docker-compose.local.yml up -d postgres
 
 # Initialize the database
@@ -44,7 +44,7 @@ pnpm inspector   # Test with MCP Inspector
 
 # File conversion (see docs/cli/)
 pnpm convert     # Convert files (mbox, eml, html, takeout)
-pnpm upload      # Upload converted markdown to Supabase
+pnpm upload      # Upload converted markdown to your knowledge base
 pnpm ui          # Web UI for conversion
 ```
 
@@ -107,7 +107,7 @@ server.tool('tool_name', {
 ```text
 src/
 ├── api/           # Express routes and middleware
-├── db/            # Database queries (Supabase)
+├── db/            # Database queries (Postgres)
 ├── services/      # Business logic (embeddings, chunking)
 ├── tools/         # MCP tool definitions
 ├── types/         # TypeScript type definitions

diff --git a/TODO.md b/TODO.md
@@ -1,162 +1,6 @@
-# Docs Sync TODO
+# Roadmap TODO
 
-Audit completed 2026-05-01. Six discrepancies found between `/docs` and the codebase. Each phase below fits within a 200k context window session.
-
----
-
-## Phase 1 — Database provider: Supabase → Neon
-
-The project migrated to Neon PostgreSQL (`DATABASE_URL`), but docs still instruct users to create a Supabase project and set `SUPABASE_URL`/`SUPABASE_SERVICE_KEY`.
-
-### Files to update
-
-**`docs/getting-started/installation.mdx`**
-
-- Lines 66–135: Replace "Option 1: Supabase (Recommended)" section with Neon setup (create project at neon.tech, copy pooled connection string into `DATABASE_URL`)
-- Lines 128–135: Replace required vars block (`SUPABASE_URL`, `SUPABASE_SERVICE_KEY`) with `DATABASE_URL=postgresql://...` and `OPENAI_API_KEY=sk-...`
-
-**`docs/getting-started/configuration.mdx`**
-
-- Lines 11–18: Rename section header "Database Connection"; replace `SUPABASE_URL`/`SUPABASE_SERVICE_KEY` rows with `DATABASE_URL` (required, Neon pooled connection string)
-- Lines 224–256: Update all three example configs (Dev, Production, Self-hosted) to use `DATABASE_URL` instead of the two Supabase vars
-
-**`docs/cli/index.mdx`**
-
-- Line 15: Change "Upload to Supabase" → "Upload to knowledge base"
-- Line 73: Change "Supabase Knowledge Base" → "Knowledge Base" in the workflow diagram
-
-**`docs/architecture/embeddings.mdx`**
-
-- Line 32: Change "Run in Supabase SQL Editor" → `psql $DATABASE_URL -f scripts/setup-db.sql`
-- Line 48: Same change for the Google AI schema instruction
-
-**`docs/mcp-tools/index.mdx`**
-
-- Line 349: Error table row "Database not configured" — change fix text from "Set `SUPABASE_URL` and `SUPABASE_SERVICE_KEY`" → "Set `DATABASE_URL`"
-- Line 160: pg-analyze section — change "independent of Supabase client" → "independent of the main database client"
-
-**`docs/guides/supabase-requirements.mdx`**
-
-- Retitle to "Database Requirements" (or rename file to `database-requirements.mdx` and update `meta.json`)
-- Replace Supabase-specific tier names/links with Neon equivalent (Neon free tier, Neon Pro, etc.)
-- Keep pgvector/HNSW sizing math — it applies to any Postgres deployment
-- Update any "Supabase Dashboard" UI references to generic psql/Neon console instructions
-
-**`docs/RUNBOOK.md`**
-
-- Update "Supabase setup" section heading and profile descriptions to reference Neon
-- Replace `SUPABASE_URL`/`SUPABASE_SERVICE_KEY` wherever they appear with `DATABASE_URL`
-
----
-
-## Phase 2 — Google AI embeddings model/dimensions + `health_check` tool
-
-Two independent issues that are small enough to tackle in one session.
-
-### 2a. Google AI model and dimensions outdated
-
-Docs say `text-embedding-004` at 768d. Codebase default is `gemini-embedding-2-preview` at 3072d.
-
-**`docs/architecture/embeddings.mdx`**
-
-- Line 14: Provider comparison table — update Google AI row: Dimensions `768` → `3072`, Model `text-embedding-004` → `gemini-embedding-2-preview`
-- Line 40–50: Google AI setup code block — change model env var default to `gemini-embedding-2-preview`, update schema reference to reflect 3072d
-- "Choose Google AI if" recommendation — update dimension mention (768 → 3072)
-
-**`docs/getting-started/configuration.mdx`**
-
-- Line 33: Table row — change `GOOGLE_EMBEDDING_MODEL` default from `text-embedding-004` → `gemini-embedding-2-preview`
-- Lines 50–53: Google AI config block — update model name and dimensions (768 → 3072)
-
-### 2b. Missing `health_check` MCP tool
-
-**`docs/mcp-tools/index.mdx`**
-
-- Line 1: Change "25 MCP tools" → "26 MCP tools" in title/description
-- Add `health_check` entry under a new "Diagnostics" section (or append to "Stats" section) with short description and link to new page
-
-**Create `docs/mcp-tools/health-check.mdx`**
-
-- Title: `health_check`
-- Description: Per-component diagnostic check (database, documents, chunks, memory, conversations, insights). Use as the first step when diagnosing infrastructure issues.
-- Parameters: none (or optional `components` filter if supported)
-- Example response: show healthy and degraded component states
-- Error handling: standard tool error format
-- Related tools: `get_stats`
-
----
-
-## Phase 3 — Missing CLI commands + model ID typo
-
-### 3a. Missing CLI converters in docs
-
-**`docs/cli/index.mdx`**
-
-- Unified Converter commands table (around line 47): Add four missing rows:
-  - `spotify <path>` — Convert Spotify data export
-  - `reddit <path>` — Convert Reddit data export
-  - `facebook <path>` — Convert Facebook data export
-  - `instagram <path>` — Convert Instagram data export
-- Available Commands section: Add `scan` and `split` entries with links to their new pages
-- Next Steps links: Add links to new `scan.mdx` and `split.mdx`
-
-**Create `docs/cli/scan.mdx`**
-
-- Title: Scan — Analyze files before upload
-- Command: `pnpm scan -- <directory-or-file> [options]`
-- What it does: inspects converted markdown files to report file sizes, estimated chunk counts, and heading structure; flags oversized files
-- Options table: `--all`, `--max-file-size <mb>`, `--max-chunks <n>`
-- Example output (tabular: file, size, estimated chunks, status)
-- Use case: run before `pnpm upload` to identify files needing splitting
-- Next steps: link to `split.mdx` and `batch-upload.mdx`
-
-**Create `docs/cli/split.mdx`**
-
-- Title: Split — Break large markdown files into chunks
-- Command: `pnpm split -- <file-or-directory> [options]`
-- What it does: splits large markdown files at heading boundaries; preserves frontmatter; adds linking metadata
-- Options table: `--split-level <h>`, `--target-chunks <n>`, `--output <dir>`, `--only-oversized`, `--dry-run`, `-r` (recursive)
-- Example: a 2000-line file split into 4 files at `##` headings
-- Next steps: link to `batch-upload.mdx`
-
-### 3b. Model ID typo
-
-**`docs/getting-started/configuration.mdx`**
-
-- Line 142: Change `EXTRACTION_MODEL=claude-haiku-4-5-20250501` → `EXTRACTION_MODEL=claude-haiku-4-5-20251001`
-
----
-
-## Phase 4 — A2A protocol docs + residual cleanup
-
-### 4a. A2A protocol documentation
-
-README lists "Agent Discovery — A2A protocol at `/.well-known/agent.json`" as a feature with no corresponding docs.
-
-**Create `docs/architecture/a2a-protocol.mdx`**
-
-- Title: Agent-to-Agent (A2A) Protocol
-- What it is: standard discovery mechanism allowing other AI agents to find and interact with this knowledge server
-- Endpoint: `GET /.well-known/agent.json` — returns agent card (name, description, capabilities, MCP endpoint)
-- How to use: point an orchestrator agent at the server URL; it auto-discovers available tools
-- Example agent card JSON
-- Security note: endpoint is publicly readable by design (no auth required), but actual tool calls still require `API_BEARER_TOKEN`
-- Related: link to MCP Tools overview
-
-**`docs/architecture/index.mdx`**
-
-- Transport layer section: add A2A as a fourth transport alongside MCP, REST, WebSocket; link to new page
-
-**`docs/getting-started/introduction.mdx`**
-
-- Key features list: add "Agent Discovery" feature entry matching README wording
-
-### 4b. Residual cleanup
-
-**`docs/mcp-tools/index.mdx`** (if not caught in Phase 1)
-
-- Confirm pg-analyze description no longer mentions "Supabase client" after Phase 1 edit
-- Confirm error table "Database not configured" row uses `DATABASE_URL` after Phase 1 edit
+The docs-sync backlog (Phases 1–4) is complete — those documentation fixes shipped and are recorded in `CHANGELOG.md` under `[0.4.0]`. What remains is the server-side Memory System Roadmap below.
 
 ---
 

diff --git a/docs/ENV.md b/docs/ENV.md
@@ -15,8 +15,8 @@ This file centralizes runtime configuration for Textrawl server, MCP tools, CLI,
 | `DATABASE_URL` | Yes (DB features) | Neon (or any PostgreSQL) pooled connection string |
 | `EMBEDDING_PROVIDER` | No | `openai` (default), `ollama`, or `google` |
 | `OPENAI_API_KEY` | Required for OpenAI | Embedding API key (`text-embedding-3-small`, 1536d) |
-| `GOOGLE_AI_API_KEY` | Required for Google | Google AI API key (`text-embedding-004`, 768d) |
-| `GOOGLE_EMBEDDING_MODEL` | No | Google embedding model (default: `text-embedding-004`) |
+| `GOOGLE_AI_API_KEY` | Required for Google | Google AI API key (`gemini-embedding-2-preview`, 3072d) |
+| `GOOGLE_EMBEDDING_MODEL` | No | Google embedding model (default: `gemini-embedding-2-preview`) |
 | `OLLAMA_BASE_URL` | Required for Ollama | Local/remote Ollama base URL |
 | `OLLAMA_MODEL` | Required for Ollama | Embedding model (e.g. `nomic-embed-text`) |
 | `API_BEARER_TOKEN` | Strongly recommended; required in prod unless OAuth | Bearer auth for `/mcp` and `/api/upload` |
@@ -40,7 +40,7 @@ This file centralizes runtime configuration for Textrawl server, MCP tools, CLI,
 | Variable | Required | Notes |
 |---|---|---|
 | `ANTHROPIC_API_KEY` | If extraction enabled | Key for `extract_memories` |
-| `EXTRACTION_MODEL` | No | Default: `claude-haiku-4-5-20250501` |
+| `EXTRACTION_MODEL` | No | Default: `claude-haiku-4-5-20251001` |
 | `INSIGHT_MODEL` | No | Default: `claude-sonnet-4-6-20250514` |
 
 ## Optional / advanced

diff --git a/docs/architecture/a2a-protocol.mdx b/docs/architecture/a2a-protocol.mdx
@@ -0,0 +1,91 @@
+---
+title: Agent-to-Agent (A2A) Protocol
+description: Agent discovery and task delegation via the A2A protocol
+---
+
+textrawl implements the **Agent-to-Agent (A2A) protocol**, a standard discovery mechanism that lets other AI agents find this knowledge server, learn its capabilities, and delegate natural-language tasks to it. This makes Textrawl usable as a knowledge backend inside a larger multi-agent system, not just as an MCP server for a single client.
+
+## Agent Card
+
+```text
+GET /.well-known/agent.json
+```
+
+A public, unauthenticated endpoint returning the agent card — a JSON description of the agent's identity, capabilities, and skills. An orchestrator agent points at the server URL and reads this card to auto-discover what Textrawl can do.
+
+### Example agent card
+
+```json
+{
+  "name": "Textrawl",
+  "description": "Personal knowledge base agent — search, store, and discover connections across your documents, memories, and conversations.",
+  "url": "https://your-server.example.com",
+  "version": "0.2.0",
+  "capabilities": {
+    "streaming": false,
+    "pushNotifications": false
+  },
+  "skills": [
+    { "id": "search", "name": "Search Knowledge", "description": "Search across all personal knowledge including documents, memories, conversations, and insights." },
+    { "id": "save", "name": "Save Knowledge", "description": "Save notes, URLs, or documents to the knowledge base." },
+    { "id": "memory", "name": "Memory Graph", "description": "Store and query facts about entities — people, projects, concepts, and their relationships." },
+    { "id": "insights", "name": "Discover Insights", "description": "Find connections and patterns across your knowledge base." }
+  ],
+  "defaultInputModes": ["text"],
+  "defaultOutputModes": ["text"],
+  "authentication": { "schemes": ["bearer"] }
+}
+```
+
+The `url` field is derived from `OAUTH_SERVER_URL` (falling back to `http://localhost:<PORT>`), so set that env var in production to advertise the correct origin.
+
+## Task Delegation
+
+```text
+POST /.well-known/agent/tasks
+```
+
+Accepts an A2A task message and routes its text through Textrawl's unified search pipeline (documents + memories + conversations, gated by `ENABLE_MEMORY` / `ENABLE_CONVERSATIONS`). Unlike the agent card, this endpoint **requires authentication** — send `Authorization: Bearer <API_BEARER_TOKEN>`.
+
+### Request
+
+```json
+{
+  "message": {
+    "parts": [{ "type": "text", "text": "What did I learn about pgvector indexing?" }]
+  }
+}
+```
+
+### Response
+
+```json
+{
+  "id": "task-1718240000000",
+  "status": { "state": "completed" },
+  "output": {
+    "parts": [{ "type": "text", "text": "[1] [document] pgvector notes: HNSW indexes trade build time for query speed..." }]
+  },
+  "metadata": {
+    "resultCount": 3,
+    "counts": { "documents": 2, "memories": 1, "conversations": 0 },
+    "sources": [{ "type": "document", "id": "550e8400-...", "title": "pgvector notes" }]
+  }
+}
+```
+
+## Security
+
+- `GET /.well-known/agent.json` is **publicly readable by design** — the agent card exposes only capability metadata, no knowledge content.
+- `POST /.well-known/agent/tasks` is protected by bearer auth. Discovery is open; actually querying your knowledge still requires `API_BEARER_TOKEN`.
+
+## How to Use
+
+1. Deploy Textrawl with `OAUTH_SERVER_URL` (or a reverse-proxy origin) and `API_BEARER_TOKEN` set.
+2. Point an orchestrator agent at the server origin; it fetches `/.well-known/agent.json` and registers the four skills.
+3. The orchestrator delegates natural-language tasks via `POST /.well-known/agent/tasks` with the bearer token.
+
+## Related
+
+- [MCP Tools Overview](/docs/mcp-tools) - The full tool surface available over MCP
+- [Architecture Overview](/docs/architecture) - Transports, request flow, and components