Skip to content

fix(fetchers): cap direct llms bodies#101

Merged
chaliy merged 1 commit intomainfrom
fix/issue-95-llms-body-size-cap
Apr 15, 2026
Merged

fix(fetchers): cap direct llms bodies#101
chaliy merged 1 commit intomainfrom
fix/issue-95-llms-body-size-cap

Conversation

@chaliy
Copy link
Copy Markdown
Contributor

@chaliy chaliy commented Apr 15, 2026

What

Make the direct llms.txt fetch path honor FetchKit's configured body-size cap and truncation semantics.

Closes #95.

Why

The direct llms.txt branch in DocsSiteFetcher read the full response body with .text().await, so it ignored max_body_size, returned oversized payloads intact, and skipped the usual truncation signal.

How

  • reuse the default fetcher's bounded body reader and shared truncation marker for direct llms.txt responses
  • apply the configured max_body_size, defaulting to the shared 10 MiB cap when unset
  • add a regression proving direct llms.txt requests truncate and report truncated: true when the cap is exceeded

Risk

  • Low
  • direct llms.txt responses now surface size and truncation metadata consistent with the default fetcher; the main regression risk is around callers that assumed uncapped bodies on this specialized path

Checklist

  • Unit tests are passed
  • Smoke tests are passed
  • Documentation is updated
  • Specs are up to date and not in conflict

@chaliy chaliy merged commit 5304095 into main Apr 15, 2026
11 checks passed
@chaliy chaliy deleted the fix/issue-95-llms-body-size-cap branch April 15, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

direct llms.txt fetch path ignores the global body size cap

1 participant