perf: reuse verified append deltas in Markstream streaming - #30
Conversation
All changes are behavior-identical (verified with differential tests against the previous build on 3,410 URL cases x2, a 76KB markdown corpus, and 4,000 random fuzz documents, plus the full 1,182-test suite). - rules/core/linkify: replace linkify-it `test()` scans over every text token with a native indexOf seed screen (`.`, `@`, `:`) — a linkify match always contains one of those characters. The seed screen is only active when every registered schema ends with `:`, otherwise the previous behavior is kept. Also drop the redundant `test()` + `match()` double scan in favor of a single `match()`. - rules/core/smartquotes: use the cached isPunctCode() instead of the two giant ucmicro regexes via isPunctChar(String.fromCharCode()) per quote. - parse/link_utils: fast-path normalizeLink/normalizeLinkText - skip punycode.toASCII/toUnicode for ASCII hostnames (identity by construction), skip mdurl.encode when every char is encode-stable (single native regex scan instead of the per-char JS loop), and skip mdurl.decode when no percent-escape sequence exists. Measured on the repo benchmark suite (50k corpora, 7 samples): - links-media-autolinks parse: ~29% faster (default config) - links_flat / autolink family fixtures: ~9-16% faster - markstream-vue-style config (html+linkify+typographer+rules) over 27 fixtures: 6.25% faster total, up to 16% on link/text-heavy fixtures - real markstream-vue getMarkdown parse (docs + chat corpus): 1-2% faster - stream append path: unchanged (within noise) - no regressions outside noise on any fixture
✅ Deploy Preview for markdown-it-ts ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Deploying markdown-it-ts with
|
| Latest commit: |
f0b852b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://dc45a4c7.markdown-it-ts.pages.dev |
| Branch Preview URL: | https://codex-markstream-performance.markdown-it-ts.pages.dev |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f0b852bf13
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| for (const sample of profile.samples) { | ||
| const id = sample | ||
| selfMs.set(id, (selfMs.get(id) || 0) + profile.timeDeltas?.[0] ?? 1) |
There was a problem hiding this comment.
For every Node .cpuprofile, this loop charges the first timeDeltas value to every sample, and the following loop then charges each sample its actual aligned delta. The resulting self and total times are therefore inflated, potentially by a large startup delta, and can produce incorrect hotspot rankings; only the aligned pass should accumulate sample time.
Useful? React with 👍 / 👎.
| const children = node.children || [] | ||
| for (const c of children) | ||
| totalMs.set(node.id, (totalMs.get(node.id) || 0) + (totalMs.get(c) || 0)) |
There was a problem hiding this comment.
In normal Node-generated profiles, parent nodes appear before their children, so this lookup reads each child's still-initialized zero total. Parents consequently report little or none of their descendants' time, despite the script sorting by total; compute totals recursively or traverse the node tree in postorder.
Useful? React with 👍 / 👎.
| const sorted = [...byName.entries()].sort((a, b) => b[1].total - a[1].total) | ||
| for (const [name, rec] of sorted.slice(0, 60)) { | ||
| console.log( | ||
| `${Math.round(rec.total).toString().padStart(8)}ms total | ${Math.round(rec.self).toString().padStart(8)}ms self | ${name}`, |
There was a problem hiding this comment.
V8 .cpuprofile timeDeltas are expressed in microseconds, but their raw sums are printed with an ms suffix here. Even after fixing the duplicate accumulation, every reported duration remains 1,000 times too large unless the accumulated values are divided by 1,000 before formatting.
Useful? React with 👍 / 👎.
| const byName = new Map() | ||
| for (let round = 0; round < rounds; round++) { | ||
| const result = spawnSync(process.execPath, ['--expose-gc', 'scripts/perf-feature-profile.mjs'], { | ||
| cwd: '/Users/Simon/Github/markdown-it-ts', |
There was a problem hiding this comment.
When the documented command is run from any checkout other than the author's machine, this hard-coded cwd does not exist, spawnSync fails, and the unchecked result is processed into an empty report. Derive the repository root from process.cwd() or import.meta.url so the committed utility can profile the checkout in which it is invoked.
Useful? React with 👍 / 👎.
| for (const name in schemas) { | ||
| if (name.charCodeAt(name.length - 1) !== 0x3A /* : */) { | ||
| safe = false |
There was a problem hiding this comment.
With the stock LinkifyIt instance, __schemas__ includes the built-in // schema, so this condition marks every default parser unsafe and mayMatch never uses the native seed filter for link-free text. That disables the main optimization described above; handle // as a known schema and include / in the seed scan while retaining the fallback for arbitrary colonless custom schemas.
Useful? React with 👍 / 👎.
| md.stream.parse(src, {}) | ||
| const t0 = performance.now() | ||
| for (let i = 0; i < segments.length; i++) | ||
| md.stream.append(segments[i], {}) |
There was a problem hiding this comment.
In stream mode, every call passes a fresh {} environment, but StreamParser.parse invalidates its cache whenever the supplied environment is not the cached object. Consequently each purported append benchmark reparses the entire accumulated source instead of exercising the append path; create one environment per bench invocation and reuse it for the initial parse and every append, or omit it consistently.
Useful? React with 👍 / 👎.
1 participant
When Markstream passes the entire accumulated Markdown on each update, the stream parser already knows the append delta but previously discarded that knowledge before tail reparsing. Reuse it to avoid repeated full-history prefix comparisons and reuse the existing tail boundary/source state. Post-block plugins continue to receive full normalized source and absolute token maps.
This also fixes five pre-existing lint errors with equivalent URL/link checks and refreshes the required performance report/fingerprint. The core-clock prototype was reverted after inconsistent consumer gains; no new API or flags are added.
Performance evidence
Full audit, reproduction, all samples and rejected experiments. Baselines: markdown-it-ts 2028739 and Markstream 1dc6f7c37. Five independent-process rounds, two warmups, Node 24.16.0 / Apple M1 Pro. These numbers isolate the underlying parser update with the old consumer unchanged:
59 workloads cover streaming, 1/8/32/97/128/211/512-character updates, full history restore, source maps/hooks, edits, CRLF, references, tables, math/HTML and real README input. Every intermediate structured output matches. Ordinary final:true restoration is outside this production optimization; its timing differences should not be attributed to append reuse. Neutral/regressing samples remain in the report.
Twelve directions were tested across both repos; three production changes survived. Table-token merging violated plugin metadata semantics; row signatures, Worker CPU, automatic AST caching, Token copying removal and core-rule analysis/clock changes did not meet the criteria. Companion Markstream PR #749 supplies the two other retained changes.
Validation