Feature/advanced markdown split - #761
Open
Naraku215 wants to merge 10 commits into
Open
Conversation
- Add atomics module for detecting and protecting tables, code blocks, math formulas, and lists during splitting - Add chunker engine with sentence-level splitting and semantic affinity based short chunk merging with continuation title cleanup - Add quality scoring module with multi-dimensional assessment - Add outline extraction and heading path tracking
- Register advanced-markdown split type in text-splitter - Add API route for split preview with chunk quality scores - Add API route for saving split results to database - Add API route for retrieving saved chunks
- Add AdvancedChunkDialog with outline tree and chunk preview - Add ChunkPreviewList with quality score display - Add OutlineTreePanel for document structure navigation - Integrate advanced split options into TaskSettings - Add split type constants and settings hook
- Add headingPath field to Chunks model in Prisma schema - Add chunk query methods in db/chunks module - Add i18n translations for advanced split features (en, zh-CN, it, pt-BR, tr)
…g in atomics - Extract beforeStart variable to avoid redundant Math.max calls and fix offset calculation in expandTableContext - Escape $ characters in atom content during restore to prevent special replacement patterns ($&, $1, etc.) from corrupting output
Naraku215
force-pushed
the
feature/advanced-markdown-split
branch
from
July 6, 2026 01:20
4e303bc to
85856ab
Compare
✅ Deploy Preview for gleeful-palmier-13a6af ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
…error logging and complete i18n - Delete 7 duplicate scoring functions from AdvancedChunkDialog.js, import from scoring.js instead - Convert scoring.js from CommonJS to ES Module exports - Replace hardcoded Chinese '所属章节：' with English 'Section:' in content generation - Add backward-compatible regex matching for both 'Section:' and '所属章节：' - Replace hardcoded Chinese '文本块' with 'Chunk' in fallback strings - Add console.error logging in catch block for summary generation failures - Add 33 missing i18n keys to it, 31 to pt-BR and tr translation files
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
变更类型
Related Issues
Closes #758
变更描述
新增 Advanced Markdown Split 分块模块，解决现有 Markdown 结构分块在处理复杂学术文献时表格被切断、公式被破坏、标题上下文丢失、产生空块等问题，并配套一套完善的质量评估和分块信息统计体系和更完善的可视化自主分块编辑操作，另外，统计信息导航重评分功能支持用户在原有的分块效果上不断优化分块质量。
做了什么：
headingPath记录其所属章节路径Dr.et al.）、小数（3.14）、省略号等边界情况变更规模： 26 files changed, +3141 / -6 lines
前端图示
关于测试
注意到项目目前尚未配置测试框架（package.json 中无 test script、无测试依赖、仓库中无现有测试文件）。本 PR 暂未添加测试用例，以避免在未与项目维护者协商的情况下引入测试框架选型（vitest / jest 等）和包管理器冲突（npm vs pnpm）。
本新增模块的所有主要测试已在本地验证通过，再如需添加测试，我可以在后续单独提交中配合项目统一配置测试基础设施。
后续建议
本模块可以看做是原markdown文档结构分块方法和自定义分块模式的组合优化，为了项目精简，建议取代这两个模块。
文档更新
English Version
Change Type
Related Issues
Closes #758
Description
Introduces the Advanced Markdown Split module to address issues with the existing Markdown structure splitting when processing complex academic documents — tables being cut, formulas broken, heading context lost, and empty chunks generated.
What this PR does:
headingPathrecording its section pathDr.,et al.), decimals (3.14), ellipsis, and mixed CJK/Latin textChange stats: 26 files changed, +3141 / -6 lines
About Tests
The project currently has no test framework configured (no test script in package.json, no test dependencies, no existing test files in the repository). This PR does not include test cases to avoid introducing a testing framework choice (vitest / jest, etc.) and package manager conflicts (npm vs pnpm) without consulting the project maintainer first.
If tests are needed, I'm happy to contribute them in a follow-up PR aligned with the project's unified test infrastructure setup.
Documentation Updates