feat(docs-site): SEO improvements — OG image, RSS feed, sitemap lastmod, structured data#508
feat(docs-site): SEO improvements — OG image, RSS feed, sitemap lastmod, structured data#508
Conversation
…friendly docs - Generate og-image.png at build time from og-image.svg via @resvg/resvg-js - Add JSON-LD structured data (WebSite + Organization schemas) to all pages - Add og:image, og:image:alt, twitter:image, robots, and author meta tags - Generate RSS 2.0 feed (feed.xml) with git-accurate publish dates per page - Generate llms-full.txt for RAG pipeline ingestion - Inject git-accurate <lastmod> dates into the Starlight-generated sitemap - Add dynamic robots.txt generation in CI (blocks old versioned paths, /main/) - Add root sitemap.xml index in CI pointing to latest version's sitemap - Copy og-image.png, llms.txt, llms-full.txt, feed.xml from latest version to root in CI - Add custom Footer component with RSS and llms.txt discovery links - Update docs-site/README.md and task6-docs.md to reflect new build/deploy behavior Closes #507
…friendly docs - Generate og-image.png at build time from og-image.svg via @resvg/resvg-js - Add JSON-LD structured data (WebSite + Organization schemas) to all pages - Add og:image, og:image:alt, twitter:image, robots, and author meta tags - Generate RSS 2.0 feed (feed.xml) with git-accurate publish dates per page - Generate llms-full.txt for RAG pipeline ingestion - Inject git-accurate <lastmod> dates into the Starlight-generated sitemap - Add dynamic robots.txt generation in CI (blocks old versioned paths, /main/) - Add root sitemap.xml index in CI pointing to latest version's sitemap - Copy og-image.png, llms.txt, llms-full.txt, feed.xml from latest version to root in CI - Add custom Footer component with RSS and llms.txt discovery links - Update docs-site/README.md and task6-docs.md to reflect new build/deploy behavior Closes #507
917be90 to
064ef5f
Compare
There was a problem hiding this comment.
Pull request overview
This PR improves SEO and content discovery for the versioned Astro/Starlight docs site (cli.internetcomputer.org) by adding social sharing assets, structured metadata, an RSS feed, git-based sitemap lastmod, and CI-generated root-level discovery files aligned with the site’s versioned deployment model.
Changes:
- Add global SEO metadata (OG/Twitter image, robots/author meta, JSON-LD structured data) and a custom footer with discovery links.
- Enhance the
astro-agent-docsbuild plugin to generatellms-full.txt,feed.xml, inject sitemaplastmodfrom git history, and renderog-image.pngfromog-image.svg. - Update CI to generate root
robots.txt+ rootsitemap.xmland copyllms-full.txt,feed.xml, andog-image.pngfrom the latest versioned build to the deployment root.
Reviewed changes
Copilot reviewed 7 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| docs-site/src/components/Footer.astro | Adds footer override with RSS + llms discovery links. |
| docs-site/public/og-image.svg | Adds source SVG for OG image generation. |
| docs-site/plugins/astro-agent-docs.mjs | Adds git-date utilities, generates llms-full.txt + RSS feed, injects sitemap lastmod, and converts SVG→PNG for OG image. |
| docs-site/package.json | Adds @resvg/resvg-js and @fontsource/inter for build-time OG PNG rendering. |
| docs-site/package-lock.json | Locks new dependencies. |
| docs-site/astro.config.mjs | Adds global head tags (RSS, robots, author, OG/Twitter, JSON-LD) and registers custom Footer component. |
| docs-site/README.md | Documents new build artifacts and root-file publishing behavior. |
| .github/workflows/docs.yml | Generates root robots.txt + sitemap.xml and copies new root discovery assets from latest version. |
| .claude/skills/release/task6-docs.md | Updates internal release instructions to reflect new docs deployment paths and root-file copy behavior. |
Files not reviewed (1)
- docs-site/package-lock.json: Language not supported
Comments suppressed due to low confidence (1)
docs-site/astro.config.mjs:15
SITEprovides a sensible default, butconfig.siteis still set fromprocess.env.PUBLIC_SITEonly. WhenPUBLIC_SITEisn’t set (e.g., localnpm run buildper the PR test plan),config.sitebecomes undefined and the agent-docs plugin will emit RSS links like/guides/...(non-canonical/invalid for many feed readers) and may also affect sitemap URL generation. Consider settingsite: SITEso local builds consistently produce absolute URLs and the plugin’ssiteUrlis populated.
export default defineConfig({
site: process.env.PUBLIC_SITE,
// For versioned deployments: /0.1/, /0.2/, etc.
// PUBLIC_BASE_PATH is set per-version in CI (e.g., /0.2/, /main/)
base: process.env.PUBLIC_BASE_PATH || '/',
markdown: {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use root-absolute URLs for feed.xml and llms.txt in head links and Footer so feed readers and agents always discover the canonical root endpoint - Use site: SITE (with fallback) instead of site: process.env.PUBLIC_SITE so siteUrl is always populated in the plugin during local builds - Add fetch-depth: 0 to all three build job checkouts so git log returns accurate dates for sitemap lastmod and RSS pubDate - Fix robots.txt contradictory Allow/Disallow /main/ when LATEST_VERSION=main - Strip BOM from per-page .md files when concatenating llms-full.txt - Memoize getGitDate() results to avoid redundant git log subprocesses - Use root-absolute siteUrl for agent signaling directive href in HTML pages
…to feat/seo-improvements # Conflicts: # .github/workflows/docs.yml # docs-site/astro.config.mjs # docs-site/plugins/astro-agent-docs.mjs # docs-site/src/components/Footer.astro
…oot files in test script
- Fix atom:link rel="self" in feed.xml to point to ${siteUrl}/feed.xml
(canonical subscription URL) instead of the versioned path
- Update test-version-switcher.sh to simulate the publish-root-files CI step:
copies llms.txt, llms-full.txt, feed.xml, og-image.png from the latest
version folder to dist-test/ root, and generates robots.txt
|
Addressed all Copilot review feedback in a follow-up commit:
|
There was a problem hiding this comment.
Pull request overview
This PR enhances the docs site’s SEO and discovery for the versioned deployment model (latest version indexed at root), adding social preview assets, RSS/sitemap improvements, and AI-agent discovery outputs.
Changes:
- Adds global SEO metadata + JSON-LD structured data and a custom Footer with discovery links.
- Extends the
astro-agent-docsbuild hook to generatellms-full.txt,feed.xml,og-image.png, and inject git-based sitemap<lastmod>. - Updates CI to generate root
robots.txt+ rootsitemap.xmland to copy root-level artifacts (llms*,feed.xml,og-image.png) from the latest versioned build.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| docs-site/test-version-switcher.sh | Mirrors CI “latest version” root redirect + root artifact generation for local multi-version testing. |
| docs-site/src/components/Footer.astro | Adds RSS + llms.txt discovery links via a Starlight Footer override. |
| docs-site/public/og-image.svg | Introduces the source SVG used to render the share image PNG at build time. |
| docs-site/plugins/astro-agent-docs.mjs | Generates llms-full.txt, RSS feed, git-based sitemap lastmod, and renders og-image.png via resvg. |
| docs-site/package.json | Adds test:versions script; adds resvg + Inter font dev dependencies. |
| docs-site/package-lock.json | Locks new dev dependencies and transitive additions. |
| docs-site/astro.config.mjs | Adds global meta tags, RSS link, JSON-LD schemas, and wires in the Footer override. |
| docs-site/README.md | Documents new build outputs and the root-file publishing pattern. |
| .github/workflows/docs.yml | Generates root robots.txt + sitemap.xml, copies additional root-level SEO/discovery files, and fetches full git history for git-date features. |
| .claude/skills/release/task6-docs.md | Updates release-process docs to reflect the new root-file copy behavior post-merge. |
Files not reviewed (1)
- docs-site/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Replace execSync shell interpolation with spawnSync args array in getGitDate to avoid shell injection on unusual file paths - Use date.slice(0, 10) for sitemap lastmod instead of toISOString() to avoid UTC conversion shifting the date for commits near midnight - Strip trailing slash from SITE constant to prevent double slashes in URLs - Clarify robots.txt comment: /main/ is conditionally disallowed, not always
|
Addressed the new (non-duplicate) findings from the second review:
Not acted on — The re-flagged comments (canonical URLs, fetch-depth, BOM, memoization, robots edge case) were already fixed in the previous follow-up — Copilot is re-reviewing the original diff lines. |
There was a problem hiding this comment.
Pull request overview
This PR improves the docs site’s SEO and “discovery” surface area in a versioned-deployment environment (latest docs served under a version path, with key SEO files published at the domain root).
Changes:
- Adds build-time generation of
og-image.png,feed.xml,llms-full.txt, and injects git-based<lastmod>into generated sitemaps. - Publishes root-level SEO/discovery files via CI (
robots.txt, rootsitemap.xml, plus copyingog-image.png/feed.xml/llms*.txtfrom the latest version folder). - Adds a Starlight Footer override and global head tags (RSS link, OG image meta, robots/author meta, JSON-LD structured data).
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| docs-site/test-version-switcher.sh | Extends local version-layout simulation to mirror CI root-file generation/copying. |
| docs-site/src/components/Footer.astro | New Footer override adding RSS + llms.txt discovery links. |
| docs-site/public/og-image.svg | Adds the source SVG used to render the social preview PNG at build time. |
| docs-site/plugins/astro-agent-docs.mjs | Generates llms-full, RSS feed, injects sitemap lastmod from git, renders OG PNG via resvg. |
| docs-site/package.json | Adds a test:versions script and new dev dependencies for OG image generation. |
| docs-site/package-lock.json | Locks new dependencies (@resvg/resvg-js, @fontsource/inter) and transitive changes. |
| docs-site/astro.config.mjs | Sets a default SITE, adds global meta/link tags + JSON-LD, wires in Footer override. |
| docs-site/README.md | Documents new build artifacts and CI root-file publishing behavior. |
| .github/workflows/docs.yml | Generates robots.txt + root sitemap.xml and copies root-level artifacts from latest version. |
| .claude/skills/release/task6-docs.md | Updates release docs to reflect updated deployment paths and root-file copying behavior. |
Files not reviewed (1)
- docs-site/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use root-relative /feed.xml and /llms.txt for head discovery links and footer links so local/staging environments don't advertise production URLs; absolute URLs are kept only for og:image/twitter:image where required - Replace CDATA with escapeXml() for RSS item descriptions to avoid invalid XML if a description contains the ]]> CDATA terminator sequence
There was a problem hiding this comment.
Pull request overview
This PR enhances SEO and content discoverability for the versioned Astro/Starlight docs deployment (root + /<version>/ folders) by adding a consistent OG image, RSS feed, structured data, robots/sitemap root files, and LLM ingestion artifacts, with CI updates to publish the correct root-level files from the latest version.
Changes:
- Add global SEO metadata + JSON-LD structured data via Starlight
headconfiguration. - Extend the
astro-agent-docsbuild hook to generatellms-full.txt,feed.xml,og-image.png, and inject git-based<lastmod>into sitemap files. - Update CI/local tooling to generate/copy root-level files (
robots.txt, rootsitemap.xml,feed.xml,llms*.txt,og-image.png) based onversions.json.
Reviewed changes
Copilot reviewed 8 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
docs-site/astro.config.mjs |
Adds global meta tags, JSON-LD, RSS/link discovery, and wires in a custom Footer override. |
docs-site/plugins/astro-agent-docs.mjs |
Generates llms-full.txt, RSS feed, injects git-based sitemap lastmod, and renders og-image.png from og-image.svg. |
.github/workflows/docs.yml |
Generates root robots.txt + root sitemap index and copies latest version root-discovery files into deployment root; ensures full git history for build jobs. |
docs-site/src/components/Footer.astro |
Adds footer discovery links for RSS and llms.txt. |
docs-site/public/og-image.svg |
Adds source SVG for OG image generation. |
docs-site/test-version-switcher.sh |
Updates local multi-version simulation to mirror new root-file behavior. |
docs-site/package.json |
Adds test:versions script; adds build-time dependencies for OG image rendering. |
docs-site/package-lock.json |
Locks newly added dependencies. |
docs-site/README.md |
Documents updated build artifacts and root-file publishing approach. |
.claude/skills/release/task6-docs.md |
Updates release process documentation for the new root-file copy behavior. |
Files not reviewed (1)
- docs-site/package-lock.json: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…od, structured data (#508) * feat(docs-site): add SEO improvements, OG image, RSS feed, and agent-friendly docs - Generate og-image.png at build time from og-image.svg via @resvg/resvg-js - Add JSON-LD structured data (WebSite + Organization schemas) to all pages - Add og:image, og:image:alt, twitter:image, robots, and author meta tags - Generate RSS 2.0 feed (feed.xml) with git-accurate publish dates per page - Generate llms-full.txt for RAG pipeline ingestion - Inject git-accurate <lastmod> dates into the Starlight-generated sitemap - Add dynamic robots.txt generation in CI (blocks old versioned paths, /main/) - Add root sitemap.xml index in CI pointing to latest version's sitemap - Copy og-image.png, llms.txt, llms-full.txt, feed.xml from latest version to root in CI - Add custom Footer component with RSS and llms.txt discovery links - Update docs-site/README.md and task6-docs.md to reflect new build/deploy behavior Closes #507 * feat(docs-site): add SEO improvements, OG image, RSS feed, and agent-friendly docs - Generate og-image.png at build time from og-image.svg via @resvg/resvg-js - Add JSON-LD structured data (WebSite + Organization schemas) to all pages - Add og:image, og:image:alt, twitter:image, robots, and author meta tags - Generate RSS 2.0 feed (feed.xml) with git-accurate publish dates per page - Generate llms-full.txt for RAG pipeline ingestion - Inject git-accurate <lastmod> dates into the Starlight-generated sitemap - Add dynamic robots.txt generation in CI (blocks old versioned paths, /main/) - Add root sitemap.xml index in CI pointing to latest version's sitemap - Copy og-image.png, llms.txt, llms-full.txt, feed.xml from latest version to root in CI - Add custom Footer component with RSS and llms.txt discovery links - Update docs-site/README.md and task6-docs.md to reflect new build/deploy behavior Closes #507 * fix(docs-site): address Copilot review feedback on SEO implementation - Use root-absolute URLs for feed.xml and llms.txt in head links and Footer so feed readers and agents always discover the canonical root endpoint - Use site: SITE (with fallback) instead of site: process.env.PUBLIC_SITE so siteUrl is always populated in the plugin during local builds - Add fetch-depth: 0 to all three build job checkouts so git log returns accurate dates for sitemap lastmod and RSS pubDate - Fix robots.txt contradictory Allow/Disallow /main/ when LATEST_VERSION=main - Strip BOM from per-page .md files when concatenating llms-full.txt - Memoize getGitDate() results to avoid redundant git log subprocesses - Use root-absolute siteUrl for agent signaling directive href in HTML pages * fix(docs-site): use canonical root URL for feed atom:self; simulate root files in test script - Fix atom:link rel="self" in feed.xml to point to ${siteUrl}/feed.xml (canonical subscription URL) instead of the versioned path - Update test-version-switcher.sh to simulate the publish-root-files CI step: copies llms.txt, llms-full.txt, feed.xml, og-image.png from the latest version folder to dist-test/ root, and generates robots.txt * docs(docs-site): mention test-version-switcher.sh in README * chore(docs-site): add test:versions npm script for version switcher testing * fix(docs-site): generate root sitemap.xml in test:versions script * fix(docs-site): point root sitemap.xml directly to sitemap-0.xml for spec compliance * fix(docs-site): address second round of Copilot review feedback - Replace execSync shell interpolation with spawnSync args array in getGitDate to avoid shell injection on unusual file paths - Use date.slice(0, 10) for sitemap lastmod instead of toISOString() to avoid UTC conversion shifting the date for commits near midnight - Strip trailing slash from SITE constant to prevent double slashes in URLs - Clarify robots.txt comment: /main/ is conditionally disallowed, not always * docs(docs-site): clarify robots.txt /main/ conditional behavior in README * fix(docs-site): address third round of Copilot review feedback - Use root-relative /feed.xml and /llms.txt for head discovery links and footer links so local/staging environments don't advertise production URLs; absolute URLs are kept only for og:image/twitter:image where required - Replace CDATA with escapeXml() for RSS item descriptions to avoid invalid XML if a description contains the ]]> CDATA terminator sequence * chore(docs-site): fix step numbering in astro-agent-docs build hook comments
Summary
Closes #507.
og-image.pnggenerated at build time frompublic/og-image.svgvia@resvg/resvg-js(Inter font, dark-mode DFINITY style). Theog:imagemeta tag always references the root URL; CI copies the image from the latest versioned build to the deployment root.WebSite+Organizationschemas injected on every page viaastro.config.mjs.og:image,og:image:alt,twitter:image,robots(index, follow, max-image-preview:large),authoradded globally.feed.xmlgenerated byastro-agent-docsplugin with git-accurate publish dates per page; linked from the new customFootercomponent and via<link rel="alternate">in<head>.lastmod: git commit dates injected into the Starlight-generated sitemap.llms-full.txt: full content dump for RAG pipeline ingestion (complementing the existingllms.txt).robots.txt(CI-generated): dynamically built inpublish-root-filesfromversions.json; allows only the latest version's path, disallows old versions and/main/(except when no releases exist yet). Not placed in versioned build output.sitemap.xml(CI-generated): sitemapindex pointing directly to/${LATEST_VERSION}/sitemap-0.xml(spec-compliant: a sitemapindex must reference sitemaps, not other sitemapindex files).publish-root-files: now also copiesog-image.png,llms-full.txt, andfeed.xmlfrom the latest versioned folder to the deployment root (same pattern as existingllms.txt).Footer.astrooverride adds RSS andllms.txtdiscovery links alongside Starlight's existing edit/pagination links.docs-site/README.mdreflects new build pipeline and root-file deployment pattern;task6-docs.mdfixes a stale path reference and notes the automatic root-file copy onversions.jsonmerge.Deployment sequence after merge
main: SEO changes are live in/main/; CI attempts to copyog-image.pngfrom the latest versioned folder (may be skipped if not yet generated there).docs/v0.2→ push:publish-versioned-docsbuilds/0.2/with the new SEO output includingog-image.png.workflow_dispatchon the docs workflow frommain(or wait for the nextmainpush):publish-root-filescopiesog-image.png,llms.txt,llms-full.txt,feed.xmlfrom0.2/to the deployment root —https://cli.internetcomputer.org/og-image.pngthen serves the new image.Test plan
cd docs-site && npm ci && npm run buildlocally — confirmdist/og-image.png,dist/llms.txt,dist/llms-full.txt,dist/feed.xmlare generatednpm run test:versions— confirm root-levelfeed.xml,llms.txt,robots.txt,sitemap.xmlare present alongside versioned foldersog-image.pngrenders correctly (dark background, ICP CLI headline)feed.xmlis valid RSS (check in a feed reader or validator)llms-full.txtcontains all page contentdocs/v0.2: verifyhttps://cli.internetcomputer.org/og-image.pngserves the new imageog:imagemeta tag appears on all pages in production🤖 Generated with Claude Code