From 38bbc4a1c86f9230e6ef88e590592981eb400437 Mon Sep 17 00:00:00 2001
From: kevin <166128744+SihanTeng@users.noreply.github.com>
Date: Fri, 7 Aug 2026 10:31:34 +0800
Subject: [PATCH] Document robots.txt controls for site owners

---
 CHANGELOG.md                     |  2 ++
 docs/content/reference/_index.md |  7 +++--
 docs/content/reference/robots.md | 53 ++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 3 deletions(-)
 create mode 100644 docs/content/reference/robots.md

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8ac50e3..8bd9768 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,8 @@ All notable changes to kage are recorded here. The format follows
 
 - `CONTRIBUTING.md` documents the development setup, test commands, and
   expectations for focused pull requests.
+- A `robots.txt` reference explains the `kage` agent token, `Crawl-delay`, and
+  the advisory `--no-robots` override ([#8](https://github.com/tamnd/kage/issues/8)).
 
 ### Fixed
 
diff --git a/docs/content/reference/_index.md b/docs/content/reference/_index.md
index 10eb292..005795f 100644
--- a/docs/content/reference/_index.md
+++ b/docs/content/reference/_index.md
@@ -6,6 +6,7 @@ weight: 30
 featured: true
 ---
 
-The exhaustive reference. The [CLI](/reference/cli/) page lists every command and
-flag; the [configuration](/reference/configuration/) page covers environment
-variables and the output layout on disk.
+The exhaustive reference. The [CLI](/reference/cli/) page lists every command
+and flag; [configuration](/reference/configuration/) covers environment
+variables and the output layout on disk; and [robots.txt](/reference/robots/)
+explains how kage applies site crawling preferences.
diff --git a/docs/content/reference/robots.md b/docs/content/reference/robots.md
new file mode 100644
index 0000000..5c4e82e
--- /dev/null
+++ b/docs/content/reference/robots.md
@@ -0,0 +1,53 @@
+---
+title: "robots.txt"
+description: "How site owners can control kage crawls, and when users can override those rules."
+weight: 30
+---
+
+kage reads `/robots.txt` before crawling and follows the group for its `kage`
+agent token by default. Agent names are matched case-insensitively, so either of
+these forms applies:
+
+```text
+User-agent: kage
+Disallow: /
+```
+
+```text
+User-agent: Kage
+Disallow: /
+```
+
+`Allow` and `Disallow` select which page paths kage may render. A disallowed
+page is skipped rather than saved for a later resumed run.
+
+## Crawl delay
+
+kage also honours the selected group's `Crawl-delay` value, spacing page-render
+starts by that duration:
+
+```text
+User-agent: kage
+Crawl-delay: 2
+```
+
+A person running the crawl can provide an explicit delay instead. This value
+takes precedence over the file:
+
+```bash
+kage clone example.com --crawl-delay 5s
+```
+
+## Advisory, not enforcement
+
+robots.txt expresses a site's crawling preference; it is not access control.
+kage users can bypass it with `--no-robots`:
+
+```bash
+kage clone example.com --no-robots
+```
+
+That flag skips the site's `Allow`, `Disallow`, and `Crawl-delay` rules. An
+explicit `--crawl-delay` is still applied. Site owners who must prevent access
+should use authentication or server-side authorization rather than relying on
+robots.txt.