spike: Serve robots.txt by sp94sap · Pull Request #21623 · SAP/spartacus

sp94sap · 2026-06-16T10:59:51Z

Problem

Requests to /robots.txt fall through to the Angular SSR engine, which returns server-rendered HTML with Content-Type: text/html. This is a hard failure for all crawlers — traditional and AI alike.

Solution

Add an explicit Express route that intercepts /robots.txt before the Angular SSR catch-all and serves a plain text response.

Changes

core-libs/setup/ssr/robots-txt/ — new module with createRobotsTxtHandler(options?), RobotsTxtOptions interface, and DEFAULT_ROBOTS_TXT_CONTENT
core-libs/setup/ssr/public_api.ts — exports new module
core-libs/schematics/src/add-ssr/files/server.ts — schematic template updated so new projects get the route wired automatically
projects/storefrontapp/src/server.ts — demo app updated
projects/ssr-tests/src/ssr-robots-txt.spec.ts — SSR E2E tests

Usage

  Default behavior — no config needed:                                                                                                                                                                                                  
  const robotsTxtHandler = createRobotsTxtHandler();        
  if (robotsTxtHandler) {                                                                                                                                                                                                               
    server.get('/robots.txt', robotsTxtHandler);                                                                                                                                                                                        
  }

Custom content override:

  createRobotsTxtHandler({                                                                                                                                                                                                              
    content: `                                              
  User-agent: *                                                                                                                                                                                                                         
  Allow: /                                                                                                                                                                                                                              
  Disallow: /checkout/                                                                                                                                                                                                
  Sitemap: https://my-storefront.example.com/sitemap.xml                                                                                                                                                                                
  `                                                                                                                                                                                                                                     
  })

Disable entirely:
createRobotsTxtHandler({ enabled: false }) // returns null, route not registered

Out of scope

Content Signals, per-bot config toggles, and CSR static asset support are deferred to follow-up tasks.

Test plan

nx run setup:test — unit tests pass
npm run test:ssr (requires built SSR app + backend) — E2E test ssr-robots-txt.spec.ts passes
Manual: curl -I http://localhost:4000/robots.txt returns Content-Type: text/plain

github-actions · 2026-06-16T11:00:12Z

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

docs: Changes to documentation only
feat: New feature work
fix: Bug fixes
perf: Code improvements for performance
refactor: Code changes that are not bug fixes or features
style: Code style changes (e.g., whitespace, formatting)
test: Adding or updating tests
chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

github-actions · 2026-06-16T11:24:28Z

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

docs: Changes to documentation only
feat: New feature work
fix: Bug fixes
perf: Code improvements for performance
refactor: Code changes that are not bug fixes or features
style: Code style changes (e.g., whitespace, formatting)
test: Adding or updating tests
chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

github-actions · 2026-06-16T11:47:03Z

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

docs: Changes to documentation only
feat: New feature work
fix: Bug fixes
perf: Code improvements for performance
refactor: Code changes that are not bug fixes or features
style: Code style changes (e.g., whitespace, formatting)
test: Adding or updating tests
chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

github-actions · 2026-06-16T12:33:55Z

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

docs: Changes to documentation only
feat: New feature work
fix: Bug fixes
perf: Code improvements for performance
refactor: Code changes that are not bug fixes or features
style: Code style changes (e.g., whitespace, formatting)
test: Adding or updating tests
chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

…engine Previously, requests to /robots.txt fell through to the Angular SSR engine and returned HTML with Content-Type: text/html, breaking all crawlers. - Adds createRobotsTxtHandler() to @spartacus/setup/ssr — returns an Express RequestHandler that serves plain text with Cache-Control headers, or null when disabled (enabled: false) - Ships DEFAULT_ROBOTS_TXT_CONTENT with rules for traditional crawlers and major AI crawlers (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Claude-SearchBot, ChatGPT-User) - Accepts RobotsTxtOptions.content for full merchant override - Route is registered before static assets and the Angular SSR catch-all - Updates the add-ssr schematic template so new projects get it wired automatically - Adds unit tests and SSR E2E test Content Signals and per-bot config toggles are out of scope for this spike. CSR-only deployments must serve robots.txt via static hosting / CDN.

github-actions · 2026-06-17T08:12:30Z

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

docs: Changes to documentation only
feat: New feature work
fix: Bug fixes
perf: Code improvements for performance
refactor: Code changes that are not bug fixes or features
style: Code style changes (e.g., whitespace, formatting)
test: Adding or updating tests
chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

github-actions · 2026-06-18T14:23:51Z

🚨 PR Title Validation Failed 🚨

Your pull request title does not follow the required format. Please update it to match the expected pattern:

Expected format:
<type>: <subject>

Allowed Types

docs: Changes to documentation only
feat: New feature work
fix: Bug fixes
perf: Code improvements for performance
refactor: Code changes that are not bug fixes or features
style: Code style changes (e.g., whitespace, formatting)
test: Adding or updating tests
chore: Build, CI, scripts, configs, etc.

Example of a valid PR title

✅ feat: Add user authentication
✅ fix: Resolve checkout bug
✅ docs: Update API documentation

❌ Merge is blocked until the PR title is corrected.

rmch91

LGTM

sp94sap requested a review from a team as a code owner June 16, 2026 10:59

github-actions Bot marked this pull request as draft June 16, 2026 11:00

sp94sap changed the title ~~feat: Serve /robots.txt from Express before Angular SSR engine (CXSPA-13220)~~ feat: Serve robots.txt Jun 16, 2026

sp94sap force-pushed the spike/ai-seo/CXSPA-13220 branch from 1d30fef to 6219cd6 Compare June 16, 2026 11:46

sp94sap force-pushed the spike/ai-seo/CXSPA-13220 branch from 6219cd6 to 5dae578 Compare June 16, 2026 12:33

sp94sap force-pushed the spike/ai-seo/CXSPA-13220 branch from 5dae578 to b438e19 Compare June 17, 2026 08:12

sp94sap changed the title ~~feat: Serve robots.txt~~ spike: Serve robots.txt Jun 18, 2026

rmch91 approved these changes Jun 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

spike: Serve robots.txt#21623

spike: Serve robots.txt#21623
sp94sap wants to merge 1 commit into
developfrom
spike/ai-seo/CXSPA-13220

sp94sap commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 17, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

rmch91 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

sp94sap commented Jun 16, 2026

Uh oh!

github-actions Bot commented Jun 16, 2026

🚨 PR Title Validation Failed 🚨

Allowed Types

Example of a valid PR title

Uh oh!

github-actions Bot commented Jun 16, 2026

🚨 PR Title Validation Failed 🚨

Allowed Types

Example of a valid PR title

Uh oh!

github-actions Bot commented Jun 16, 2026

🚨 PR Title Validation Failed 🚨

Allowed Types

Example of a valid PR title

Uh oh!

github-actions Bot commented Jun 16, 2026

🚨 PR Title Validation Failed 🚨

Allowed Types

Example of a valid PR title

Uh oh!

github-actions Bot commented Jun 17, 2026

🚨 PR Title Validation Failed 🚨

Allowed Types

Example of a valid PR title

Uh oh!

github-actions Bot commented Jun 18, 2026

🚨 PR Title Validation Failed 🚨

Allowed Types

Example of a valid PR title

Uh oh!

rmch91 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants