Skip to content

Commit 1d95a53

Browse files
authored
feat(llms): add auto-generate llms.txt for llms support (#162)
2 parents cdc6859 + 9ffefc5 commit 1d95a53

File tree

12 files changed

+205
-31
lines changed

12 files changed

+205
-31
lines changed

.github/workflows/auto-release.yml

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,6 @@ jobs:
7070
## Full Changelog
7171
[View Full Changelog](https://github.com/ZL-Asica/SuzuBlog/blob/main/CHANGELOG.md)
7272
73-
## Contributors
74-
🏆 Thank you to everyone who contributed to this release!
75-
7673
${{ github.event.release.body }}
7774
draft: false
7875
prerelease: false

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,6 @@ yarn-error.log*
3737

3838
# RSS generated by Suzublog
3939
/public/feed.xml
40+
# llms.txt & llms-full.txt generated by Suzublog
41+
/public/llms.txt
42+
/public/llms-full.txt

CHANGELOG.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,19 @@
11
# SuzuBlog Changelog
22

3+
## 1.9.0 (2025-04-23)
4+
5+
### Minor Changes
6+
7+
- Add auto-generation for llms.txt & llms-full.txt
8+
9+
- Add auto-generation for `llms.txt` and `llms-full.txt` files at build time.
10+
- Followed the guidelines on [llmstxt.org](https://llmstxt.org/) to generate the files.
11+
- This allows real time MCP, RAG, or LLM model doing web searching by itself could better understand the context of the whole website.
12+
- `llms.txt` contains basic information for the whole website include some links.
13+
- `llms-full.txt` contains all the content of the website, include full posts contents, about, friends.
14+
- Fix `robots.txt` to allow search engines to crawl some files in `_next` folder.
15+
- This will fix crawlers are not able to render the website properly (since no css and js files are loaded).
16+
317
## 1.8.2 (2025-04-21)
418

519
### Patch Changes

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
- **🌓 Dark Mode** – Adapts to system preferences seamlessly.
2121
- **📢 RSS Feed** – Auto-generated RSS for easy content distribution.
2222
- **♿ Accessibility First** – Semantic HTML, ARIA support, WCAG-compliant colors.
23+
- **⚛️ LLM Support** – Auto-generated `llms.txt` and `llms-full.txt` files for LLMs like ChatGPT, Claude, and more.
2324

2425
## **🚀 Get Started**
2526

@@ -37,7 +38,7 @@ For setup, configuration, Markdown syntax, and deployment guides, follow the doc
3738
├── public # Static assets directory
3839
│ └── images # Image resources
3940
├── src # Project source code
40-
│ ├── app # Next.js application directory
41+
│ ├── app # Next.js App Router
4142
│ ├── components # Reusable components
4243
│ ├── services # Logic for content parsing, configuration, etc.
4344
│ └── types.d.ts # Global type definitions

README_JA.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
- **🌓 ダークモード** – システム設定に応じて自動でテーマを切り替え。
2121
- **📢 RSS フィード** – 自動生成される RSS でブログの更新を簡単に配信。
2222
- **♿ アクセシビリティ対応** – セマンティック HTML、ARIA サポート、WCAG 基準のカラーデザイン。
23+
- **⚛️ LLM 対応**`llms.txt``llms-full.txt` を自動生成し、ChatGPT や Claude などの LLM に対応。
2324

2425
## 🚀 はじめに
2526

@@ -37,10 +38,10 @@ Suzu Blog のセットアップ、設定、Markdown の書き方、デプロイ
3738
├── public # 静的リソースディレクトリ
3839
│ └── images # 画像リソース
3940
├── src # プロジェクトソースコード
40-
│ ├── app # Next.js ページディレクトリ
41+
│ ├── app # Next.js App Router
4142
│ ├── components # 再利用可能なコンポーネント
4243
│ ├── services # コンテンツ解析、設定処理などのロジック
43-
│ └── types.d.ts # グローバル型定義
44+
│ └── types # グローバル型定義
4445
├── package.json # プロジェクト依存関係とスクリプト
4546
└── pnpm-lock.yaml # pnpm 依存関係ロックファイル
4647
```

README_ZH.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
- **🌓 深色模式** – 自动适配系统主题,无缝切换。
2121
- **📢 RSS 订阅** – 自动生成 RSS,方便订阅和分发内容。
2222
- **♿ 无障碍优化** – 语义化 HTML、ARIA 支持、符合 WCAG 规范的色彩设计。
23+
- **⚛️ LLM 支持** – 自动生成 `llms.txt``llms-full.txt` 文件,兼容 ChatGPT、Claude 等 LLM。
2324

2425
## 🚀 快速上手
2526

@@ -37,10 +38,10 @@ Suzu Blog 的安装、配置、Markdown 语法、部署等详细教程,请参
3738
├── public # 静态资源目录
3839
│ └── images # 图片资源
3940
├── src # 项目源代码
40-
│ ├── app # Next.js 页面目录
41+
│ ├── app # Next.js App Router
4142
│ ├── components # 复用组件
4243
│ ├── services # 服务逻辑(内容解析、配置加载等)
43-
│ └── types.d.ts # 全局类型定义
44+
│ └── types # 全局类型定义
4445
├── package.json # 项目依赖与脚本
4546
└── pnpm-lock.yaml # pnpm 依赖锁定
4647
```

package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "suzu-blog",
3-
"version": "1.8.2",
3+
"version": "1.9.0",
44
"private": true,
55
"packageManager": "[email protected]",
66
"author": {

src/app/[slug]/page.tsx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,8 @@
11
import type { Metadata } from 'next'
22
import { ArticlePage } from '@/components/article'
33
import { getConfig } from '@/services/config'
4-
54
import { getAllPosts, getPostData } from '@/services/content'
6-
import generateRssFeed from '@/services/utils/generateRssFeed'
5+
import { generateLLMsTXTs, generateRssFeed } from '@/services/utils'
76
import Head from 'next/head'
87

98
import { notFound, redirect } from 'next/navigation'
@@ -15,6 +14,7 @@ export async function generateStaticParams() {
1514
if (config.socialMedia.rss !== null) {
1615
await generateRssFeed(posts, config)
1716
}
17+
await generateLLMsTXTs(posts, config)
1818
return posts.map(post => ({
1919
slug: post.slug,
2020
}))

src/app/robots.ts

Lines changed: 8 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,38 +1,28 @@
11
import type { MetadataRoute } from 'next'
22

33
import { getConfig } from '@/services/config'
4-
import { getAllPosts } from '@/services/content'
54

6-
async function robots(): Promise<MetadataRoute.Robots> {
5+
function robots(): MetadataRoute.Robots {
76
const config = getConfig()
87
const siteUrl = config.siteUrl
98

10-
// Generate robots.txt entries for each post
11-
const posts = await getAllPosts()
12-
const postUrls = posts.map(post => `/${post.slug}`)
13-
149
// Pages settings
15-
const showAnime = config.anilist_username === undefined || config.anilist_username !== null || config.anilist_username !== ''
10+
const showAnime = Boolean(config.anilist_username?.trim())
1611

1712
const allowList = [
1813
'/',
19-
'/about',
20-
'/friends',
21-
'/posts',
22-
...postUrls, // Dynamic post URLs
14+
'/_next/static/css',
15+
'/_next/image',
16+
'/_next/static/media',
17+
'/_next/static/chunks',
2318
]
2419

2520
const disallowList = [
26-
'/posts?',
27-
'/images',
28-
'/icons',
21+
'/api',
2922
'/_next',
3023
]
3124

32-
if (showAnime) {
33-
allowList.push('/about/anime')
34-
}
35-
else {
25+
if (!showAnime) {
3626
disallowList.push('/about/anime')
3727
}
3828

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
'use server'
2+
3+
import { promises as fs } from 'node:fs'
4+
import path from 'node:path'
5+
import process from 'node:process'
6+
import { getPostData } from '../content'
7+
8+
// Utility to clean frontmatter delimiters and adjust heading levels
9+
const cleanFrontmatter = (raw: string): string =>
10+
raw.trim().replace(/---\n/g, '').replace(/\n## /g, '\n### ')
11+
12+
// Build the About section for both markdown and llms-full contents
13+
const buildAboutSection = async (): Promise<string> => {
14+
let llms = '---\n/about\n---\n\n'
15+
16+
try {
17+
const data = await getPostData('About')
18+
if (data !== null) {
19+
const content = cleanFrontmatter(data.contentRaw)
20+
llms += `# About\n\n${content}`
21+
return llms
22+
}
23+
}
24+
catch {
25+
// Fallback content
26+
}
27+
28+
const fallback = 'This is a personal blog with technical articles and thoughts.'
29+
llms += `# About\n\n${fallback}`
30+
return llms
31+
}
32+
33+
// Build the Posts section
34+
const buildPostsSection = (
35+
posts: PostListData[],
36+
siteUrl: string,
37+
): { md: string, llms: string } => {
38+
const mdLines: string[] = ['## Posts', '']
39+
const llmsParts: string[] = []
40+
41+
posts.forEach((post) => {
42+
const url = `${siteUrl}/${post.slug}`
43+
const { title, date, redirect } = post.frontmatter
44+
const abstract = post.postAbstract?.replace(/\n/g, ' ') ?? ''
45+
46+
mdLines.push(`- [${title}](${url}): ${abstract} (${date})`)
47+
llmsParts.push(`---\n${url}\n---\n`)
48+
49+
const redirected = Boolean(redirect?.trim())
50+
51+
if (redirected) {
52+
llmsParts.push(
53+
`# ${title} (Redirect)\n\nThis post is a redirect to ${redirect}`,
54+
)
55+
}
56+
else {
57+
const content = post.contentRaw.trim().replace(/---\n/g, '')
58+
llmsParts.push(`#${title}\n\n${content}`)
59+
}
60+
})
61+
62+
return { md: mdLines.join('\n'), llms: llmsParts.join('\n') }
63+
}
64+
65+
// Build Friends & Posts archive section
66+
const buildFriendsSection = async (
67+
siteUrl: string,
68+
author: string,
69+
): Promise<{ md: string, llms: string }> => {
70+
const md = [
71+
`- [About](${siteUrl}/about): About page with personal information for ${author}`,
72+
`- [Friends](${siteUrl}/friends): A curated list of linked blogs and friend sites`,
73+
`- [All Posts](${siteUrl}/posts): Chronological listing of all blog entries`,
74+
].join('\n')
75+
76+
let llms = '---\n/friends\n---\n\n'
77+
try {
78+
const data = await getPostData('Friends')
79+
if (data !== null) {
80+
const content = data.contentRaw.trim().replace(/---\n/g, '')
81+
llms += `# Friends\n\n${content}`
82+
return { md, llms }
83+
}
84+
}
85+
catch {
86+
// Fallback
87+
}
88+
89+
llms += '# Friends\n\nA curated list of linked blogs and friend sites.'
90+
return { md, llms }
91+
}
92+
93+
// Main function to generate llms.txt and llms-full.txt
94+
const generateLLMsTXTs = async (
95+
posts: PostListData[],
96+
config: Config,
97+
): Promise<void> => {
98+
const { siteUrl, title, subTitle, description, author, slogan, anilist_username } = config
99+
const showAnime = Boolean(anilist_username?.trim())
100+
101+
// Header
102+
const headerMd = [
103+
`# ${title} - ${subTitle}`,
104+
'',
105+
`> ${description ?? 'Another SuzuBlog based personal blog.'} - ${author.name}. Personal slogan: ${slogan}`,
106+
'',
107+
`This website is using SuzuBlog, which is developed by [ZL Asica](https://zla.pub/) (she/her) based on Next.js.`,
108+
'',
109+
].join('\n')
110+
111+
let markdownContent = `${headerMd}\n`
112+
const llmsContents: string[] = []
113+
114+
// About
115+
const about = await buildAboutSection()
116+
llmsContents.push(about)
117+
118+
// Posts
119+
const postsSection = buildPostsSection(posts, siteUrl)
120+
markdownContent += `\n${postsSection.md}\n`
121+
llmsContents.push(postsSection.llms)
122+
123+
// Friends & archive
124+
const friendsSection = await buildFriendsSection(siteUrl, config.author.name)
125+
markdownContent += `\n## Optional\n\n${friendsSection.md}\n`
126+
llmsContents.push(friendsSection.llms)
127+
128+
// Archive
129+
llmsContents.push(
130+
['---', '/posts', '---', '', '# Posts', '', 'Chronological listing of all blog entries'].join('\n'),
131+
)
132+
133+
// Anime
134+
if (showAnime) {
135+
markdownContent += `- [Anime](${siteUrl}/about/anime): Automatically updated anime watch list from AniList\n`
136+
llmsContents.push(
137+
['---', '/about/anime', '---', '', '# Anime', '', 'Automatically updated anime watch list from AniList'].join('\n'),
138+
)
139+
}
140+
141+
// Write to disk
142+
try {
143+
const outputDir = path.join(process.cwd(), 'public')
144+
await fs.mkdir(outputDir, { recursive: true })
145+
await fs.writeFile(path.join(outputDir, 'llms.txt'), markdownContent, 'utf8')
146+
await fs.writeFile(
147+
path.join(outputDir, 'llms-full.txt'),
148+
llmsContents.join('\n\n'),
149+
'utf8',
150+
)
151+
// eslint-disable-next-line no-console
152+
console.info('llms.txt & llms-full.txt generated in /public 🎉')
153+
}
154+
catch (err) {
155+
console.error(
156+
'Failed to write LLMs files:',
157+
err instanceof Error ? err.message : err,
158+
)
159+
}
160+
}
161+
162+
export default generateLLMsTXTs

0 commit comments

Comments
 (0)