diff --git a/features/crawl.mdx b/features/crawl.mdx index e528b20b..c1cd8621 100644 --- a/features/crawl.mdx +++ b/features/crawl.mdx @@ -259,6 +259,7 @@ The full set of parameters available when submitting a crawl job: | `allowExternalLinks` | `boolean` | `false` | Follow links to external websites | | `sitemap` | `string` | `"include"` | Sitemap handling: `"include"` (default), `"skip"`, or `"only"` | | `ignoreQueryParameters` | `boolean` | `false` | Avoid re-scraping the same path with different query parameters | +| `ignoreRobotsTxt` | `boolean` | `false` | Ignore the website's robots.txt rules when crawling. Requires account-level approval — contact support to enable this feature. | | `delay` | `number` | (none) | Delay in seconds between scrapes to respect rate limits. Setting this forces concurrency to 1. | | `maxConcurrency` | `integer` | (none) | Maximum concurrent scrapes. Defaults to your team's concurrency limit. | | `scrapeOptions` | `object` | (none) | Options applied to every scraped page (formats, proxy, caching, actions, etc.) |