Document robots.txt controls for site owners

CHANGELOG.md

@@ -10,6 +10,8 @@ All notable changes to kage are recorded here. The format follows
 
 - `CONTRIBUTING.md` documents the development setup, test commands, and
   expectations for focused pull requests.
+- A `robots.txt` reference explains the `kage` agent token, `Crawl-delay`, and
+  the advisory `--no-robots` override ([#8](https://github.com/tamnd/kage/issues/8)).
 
 ### Fixed
 

docs/content/reference/_index.md

@@ -6,6 +6,7 @@ weight: 30
 featured: true
 ---
 
-The exhaustive reference. The [CLI](/reference/cli/) page lists every command and
-flag; the [configuration](/reference/configuration/) page covers environment
-variables and the output layout on disk.
+The exhaustive reference. The [CLI](/reference/cli/) page lists every command
+and flag; [configuration](/reference/configuration/) covers environment
+variables and the output layout on disk; and [robots.txt](/reference/robots/)
+explains how kage applies site crawling preferences.

docs/content/reference/robots.md

@@ -0,0 +1,53 @@
+---
+title: "robots.txt"
+description: "How site owners can control kage crawls, and when users can override those rules."
+weight: 30
+---
+
+kage reads `/robots.txt` before crawling and follows the group for its `kage`
+agent token by default. Agent names are matched case-insensitively, so either of
+these forms applies:
+
+```text
+User-agent: kage
+Disallow: /
+```
+
+```text
+User-agent: Kage
+Disallow: /
+```
+
+`Allow` and `Disallow` select which page paths kage may render. A disallowed
+page is skipped rather than saved for a later resumed run.
+
+## Crawl delay
+
+kage also honours the selected group's `Crawl-delay` value, spacing page-render
+starts by that duration:
+
+```text
+User-agent: kage
+Crawl-delay: 2
+```
+
+A person running the crawl can provide an explicit delay instead. This value
+takes precedence over the file:
+
+```bash
+kage clone example.com --crawl-delay 5s
+```
+
+## Advisory, not enforcement
+
+robots.txt expresses a site's crawling preference; it is not access control.
+kage users can bypass it with `--no-robots`:
+
+```bash
+kage clone example.com --no-robots
+```
+
+That flag skips the site's `Allow`, `Disallow`, and `Crawl-delay` rules. An
+explicit `--crawl-delay` is still applied. Site owners who must prevent access
+should use authentication or server-side authorization rather than relying on
+robots.txt.