From 285632e0b38b5fc707cf3863c7a962e7e64f5a5d Mon Sep 17 00:00:00 2001 From: "firecrawl-spring[bot]" <254786068+firecrawl-spring[bot]@users.noreply.github.com> Date: Tue, 7 Apr 2026 02:05:53 +0000 Subject: [PATCH] docs(crawl): add ignoreRobotsTxt parameter to configuration reference Co-Authored-By: micahstairs --- features/crawl.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/features/crawl.mdx b/features/crawl.mdx index e528b20b..c1cd8621 100644 --- a/features/crawl.mdx +++ b/features/crawl.mdx @@ -259,6 +259,7 @@ The full set of parameters available when submitting a crawl job: | `allowExternalLinks` | `boolean` | `false` | Follow links to external websites | | `sitemap` | `string` | `"include"` | Sitemap handling: `"include"` (default), `"skip"`, or `"only"` | | `ignoreQueryParameters` | `boolean` | `false` | Avoid re-scraping the same path with different query parameters | +| `ignoreRobotsTxt` | `boolean` | `false` | Ignore the website's robots.txt rules when crawling. Requires account-level approval — contact support to enable this feature. | | `delay` | `number` | (none) | Delay in seconds between scrapes to respect rate limits. Setting this forces concurrency to 1. | | `maxConcurrency` | `integer` | (none) | Maximum concurrent scrapes. Defaults to your team's concurrency limit. | | `scrapeOptions` | `object` | (none) | Options applied to every scraped page (formats, proxy, caching, actions, etc.) |