Better mini sitemap for the chatbot

.github/workflows/validate-docs-site.yaml

@@ -184,10 +184,26 @@ jobs:
         sudo apt-get update
         sudo apt-get install -y pandoc
 
+    - name: Verify chatbot product map is up to date
+      run: |
+        python3 site/scripts/generate_chatbot_product_map.py
+        git diff --exit-code site/llm/chatbot-product-map.md site/llm/chatbot-product-map-frontend-snapshot.json
+
+    - name: Test chatbot product map generator
+      run: python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py' -v
+
     - name: Validate LLM markdown render
       run: bash llm/render.sh && bash llm/clean.sh
       working-directory: site
 
+    - name: Verify LLM corpus includes product map and docs IA hub
+      run: |
+        test -f site/llm/_llm-output/chatbot-product-map.md
+        test -f site/llm/_llm-output/AGENTS.md
+        test -f site/llm/_llm-output/about/contributing/using-the-documentation.md
+        test ! -f site/llm/_llm-output/about/contributing/validmind-community.md
+        test ! -d site/llm/_llm-output/about/contributing/style-guide
+
     # Release headroom and shrink before final lightweight steps & post-job
     - name: Release reserve & shrink
       if: always()

AGENTS.md

@@ -32,6 +32,16 @@ If you are an AI agent embedded in ValidMind, your capabilities are documented h
 
 This page describes what the assistant can and cannot do, including context-aware features and current limitations.
 
+## Product UI mapping
+
+The in-app assistant (Valerie) also ingests **`chatbot-product-map.md`** in the LLM corpus. That file maps **platform routes** (for example `/settings/workflows`, `/model-inventory`, `/dashboard`) to documentation URLs and section hints.
+
+Route and help-link data from the product UI is vendored as **`site/llm/chatbot-product-map-frontend-snapshot.json`**. Regenerate it with `make -C site refresh-chatbot-product-map` when frontend routes or `helpLink` values change (requires a local `validmind/frontend` checkout).
+
+Use the map when the user’s question is tied to where they are in the product — especially **Settings**, where the UI groups features differently than the documentation sidebars (Configuration, Workflows, Inventory, and so on).
+
+For documentation organized by topic, continue to use **Using the documentation** (above) and the section table in this file.
+
 ## File format
 
 Documentation is written in Quarto Markdown (`.qmd`). Key conventions:

site/Makefile

@@ -13,7 +13,7 @@ SRC_ROOT := _source
 SRC_DIR := $(SRC_ROOT)/validmind-library
 
 # Define .PHONY target for help section
-.PHONY: help add-copyright clean clone copy-installation copy-release-notes delete-demo-branch deploy-demo-branch deploy-prod deploy-staging docker-build docker-serve docker-site docker-site-lite docs-site execute generate-sitemap get-api-json get-source kind-serve kind-stop kind-restart kind-logs notebooks python-docs release-notes render-llm template-schema-docs verify-copyright yearly-releases
+.PHONY: help add-copyright clean clone copy-installation copy-release-notes delete-demo-branch deploy-demo-branch deploy-prod deploy-staging docker-build docker-serve docker-site docker-site-lite docs-site execute generate-chatbot-product-map refresh-chatbot-product-map generate-sitemap get-api-json get-source kind-serve kind-stop kind-restart kind-logs notebooks python-docs release-notes render-llm template-schema-docs verify-copyright yearly-releases
 
 # Help section
 help:
@@ -38,6 +38,8 @@ help:
 	@echo "  docker-site              Get source, render site with Docker profile, execute notebooks"
 	@echo "  docker-site-lite         Get source and render site with Docker profile (skips notebook execution)"
 	@echo "  docs-site                Get all source files and render the production docs site with Quarto"
+	@echo "  generate-chatbot-product-map  Generate product-to-docs map (from committed frontend snapshot)"
+	@echo "  refresh-chatbot-product-map   Re-extract frontend snapshot + map (requires ../frontend)"
 	@echo "  generate-sitemap         Generate a sitemap for the static HTML site"
 	@echo "  execute                  Execute a Jupyter Notebook or notebook directory"
 	@echo "  get-api-json             Download Swagger JSON specs from ValidMind APIs into reference/"
@@ -439,6 +441,14 @@ yearly-releases:
 	git status | grep -v 'release-scripts/'
 	quarto preview
 
+# Generate product-to-documentation map for chatbot RAG (uses committed frontend snapshot)
+generate-chatbot-product-map:
+	@python3 scripts/generate_chatbot_product_map.py
+
+# Refresh vendored frontend snapshot from a local validmind/frontend checkout
+refresh-chatbot-product-map:
+	@python3 scripts/generate_chatbot_product_map.py --from-frontend
+
 # Render site to GFM markdown for LLM ingestion
 render-llm:
 	@echo "\nRendering site to GFM markdown for LLM ingestion ..."

site/llm/README.md

@@ -0,0 +1,54 @@
+# LLM corpus (markdown for RAG)
+
+This directory holds tooling that renders the documentation site to plain Markdown for ingestion into LanceDB and the in-app assistant (Valerie).
+
+## Output
+
+Rendered files land in `site/llm/_llm-output/` (gitignored). CI publishes that directory after `make render-llm`. Deployed environments only pick up new assistant context after the LanceDB artifact is rebuilt from a fresh render.
+
+## Render locally
+
+From `site/`:
+
+```bash
+make render-llm
+```
+
+This runs `llm/render.sh` (temporary minimal `_quarto.yml`, Quarto → GFM) and `llm/clean.sh` (Pandoc cleanup). Equivalent to the **Validate LLM markdown render** step in `.github/workflows/validate-docs-site.yaml`.
+
+Excluded from the LLM render (among others): notebooks, internal pages, contributor style guide, and most of `about/contributing/`. **Included:** `about/contributing/using-the-documentation.qmd` (docs IA hub for agents).
+
+Copied into `_llm-output/` after render:
+
+| File | Source |
+|------|--------|
+| `AGENTS.md` | Repo root |
+| `chatbot-product-map.md` | Generated (see below) |
+| `about/contributing/using-the-documentation.md` | Quarto render |
+
+See also [`AGENTS.md`](../../AGENTS.md) for how agents should use the corpus.
+
+## Chatbot product map
+
+Valerie needs routes in the product UI (especially **Settings**) mapped to documentation URLs. Human docs sidebars are organized by topic (Configuration, Workflows, Inventory); the product groups features differently.
+
+| Artifact | Purpose |
+|----------|---------|
+| `chatbot-product-map-frontend-snapshot.json` | Vendored extract from `validmind/frontend` (Settings tree, sidebar nav, `helpLink` / docs URLs) |
+| `chatbot-product-map.md` | Retrieval-oriented map: routes → primary/related docs + section headings from `.qmd` |
+
+### Why a frontend snapshot?
+
+CI does **not** check out `validmind/frontend` (private repo; cross-repo PAT scope). The snapshot is committed in this repo so pipelines can regenerate the map without frontend access. It may lag the live product until someone refreshes it locally.
+
+### Maintenance
+
+| Change | Command (from `site/`) |
+|--------|-------------------------|
+| Docs only (new `.qmd`, heading updates) | `make generate-chatbot-product-map` |
+| Product routes or in-app help links | `make refresh-chatbot-product-map` (requires a sibling `../frontend` checkout) |
+
+Commit both `chatbot-product-map.md` and `chatbot-product-map-frontend-snapshot.json` when the snapshot changes. CI fails if either file is out of date after regeneration.
+
+Generator: `site/scripts/generate_chatbot_product_map.py`  
+Tests: `python3 -m unittest discover -s site/scripts -p 'test_generate_chatbot_product_map.py' -v` (from repo root).

site/llm/_quarto.yml

@@ -12,7 +12,8 @@ project:
     - "../**/*.qmd"
     - "!../notebooks/"
     - "!../404.qmd"
-    - "!../about/contributing/"
+    - "!../about/contributing/validmind-community.qmd"
+    - "!../about/contributing/style-guide/"
     - "!../about/deployment/"
     - "!../about/fine-print/"
     - "!../_site/"