feat: ingest prematch optimization#516
Open
yudaiyan wants to merge 29 commits into
Open
Conversation
yudaiyan
force-pushed
the
feat/index-optimization
branch
from
July 10, 2026 08:51
74bb987 to
b24aac5
Compare
- Remove literal '---FILE: wiki/index.md---' from prompt that taught
the model the exact pattern to output (classic prompt anti-pattern)
- Add parser defense: silently skip wiki/index.md FILE blocks to
prevent overwriting index.md (which is updated via INDEX blocks)
- Add reasoning: { mode: 'off' } to prematch LLM calls, consistent
with all other ingest pipeline calls
- Skip wiki/overview.md FILE blocks in parser (Task 6) - Replace overview FILE block instruction with OVERVIEW block format in prompt (Task 7) - Add overview prematch (Step 0.8) and OVERVIEW block processing (Step 3.6) (Task 8) - Remove overview.md from aggregate repair (Task 9)
stableContextLength now uses reducedIndex/reducedOverview lengths instead of full index/overview. This prevents over-trimming the source budget when prematch has already significantly reduced the context.
DeepSeek V4 Flash supports up to 1M token context. The previous 64K char setting caused computeIngestGenerationMaxTokens to return the minimum 8192 tokens, truncating generation output.
- Increase CHUNK_SIZE from 50 to 200 entries per chunk - Increase overview prematch chunk from 2000 to 8000 chars - Remove max_tokens: 256 limit from prematch calls (let model decide) - Remove Math.min(8_192) cap from computeIngestReviewMaxTokens
Previously prematch was skipped when index/overview fit in one chunk (<=200 entries or <=8000 chars). This meant those entries were never filtered, wasting context. Now prematch runs for any chunked content, reducing even small indexes to only relevant entries.
Change overview prematch from ##-section granularity to paragraph-level. Each paragraph (split by \n\n within a ## section) is individually numbered and matched by the LLM. assembleReducedOverview groups matched paragraphs under their section headings for generation prompt injection. - Add parseOverviewParagraphs helper - Update chunkOverviewBySections to number paragraphs globally - Update buildOverviewPrematchPrompt for paragraph numbers - Update parseOverviewPrematchOutput to return number[] - Update runOverviewPrematchParallel return type - Update assembleReducedOverview to accept paragraph numbers - Update ingest.ts call site variable names and log messages - Update scenario tests to include prematch mock response
- Index/overview prematch: reasoning max for precise matching - Index chunk size: 200 → 500 entries per chunk - Overview chunk size: 8000 → 16000 chars per chunk - Add INGEST_GENERATION_TOKENS_1M = 384000 for 1M context models - computeIngestReviewMaxTokens: remove /2 cap - Analysis and long-source: max_tokens 4096 → computeIngestReviewMaxTokens - Review prompt: 'output nothing' → 'output exactly: none' - Overview prematch prompt: 'related to' → 'would need to be UPDATED' - DeepSeek V4: thinking=enabled for reasoning max, reasoning_effort for high/max - Tests updated for new token limits and DeepSeek flash thinking behavior
Aggregate repair only writes log.md now (index/overview are updated incrementally via INDEX/OVERVIEW blocks in the generation step). - Pass empty strings instead of full index (125K chars) and full overview (185K chars) as context - Remove prompt instructions for generating complete index/overview
…educedIndex - Add detectCodeReviewItems: pure-code missing-page/duplicate detection by scanning generation output wikilinks against wiki index and current FILE blocks. 100% accurate, 0 LLM tokens. - LLM prompt narrowed to suggestion + contradiction only (no longer asks about missing-page or duplicate). - Pass reducedIndex (prematch-filtered) instead of full wiki index to the LLM. Preserves cross-page suggestion capability at minimal cost. - Review input drops from ~105K tokens to ~12K tokens (89% reduction).
DeepSeek API only supports high/max for reasoning_effort. Using high instead of max reduces thinking cost while maintaining accuracy for prematch discrimination. Non-prematch stages keep off.
When reasoning=high, the model may produce reasoning_content but no content — meaning it concluded no paragraphs match. Treat this as a valid empty result instead of logging a warning.
- Prematch prompt: tell model not to match paragraphs that already cover the topic adequately — re-ingesting the same topic does not require an update. - Generation OVERVIEW instruction: if the source only restates information already in the overview context, output NO OVERVIEW block. - Remove leftover text from previous incomplete revert.
yudaiyan
force-pushed
the
feat/index-optimization
branch
from
July 14, 2026 06:08
b24aac5 to
d8be245
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add index and overview pre-match stages to the ingest pipeline, plus downstream optimizations for review-suggestion and aggregate-repair stages.
Changes
Prematch optimization
sourceBudget.DeepSeek V4 thinking support
thinking: { type: enabled/disabled }withreasoning_effortfor high/max modes.reasoning=highfor accurate relevance discrimination.reasoning=offto avoid thinking overhead on content generation.Pipeline stage optimizations
detectCodeReviewItems— scans generation output wikilinks against wiki index). LLM call now only handles suggestion + contradiction, usingreducedIndexinstead of full index. Input drops from ~105K to ~12K tokens.Token limit improvements
INGEST_GENERATION_TOKENS_1M = 384_000for 1M context models.computeIngestReviewMaxTokens: removed /2 division cap.suggestedContextSizechanged from 64K to 1,000,000.Incremental index/overview
---INDEX: Category---blocks instead of complete index.md. Parsed and incrementally appended.Error handling
none.