Skip to content

*: make low-latency mode changefeed-scoped - #5862

Draft
asddongmen wants to merge 21 commits into
pingcap:masterfrom
asddongmen:agent/changefeed-level-low-latency-mode
Draft

*: make low-latency mode changefeed-scoped#5862
asddongmen wants to merge 21 commits into
pingcap:masterfrom
asddongmen:agent/changefeed-level-low-latency-mode

Conversation

@asddongmen

@asddongmen asddongmen commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #5705

TiCDC's low-latency optimizations were configured per server, so throughput
and latency-sensitive changefeeds could not safely coexist on the same
captures. Resolved-ts observability and high-cardinality EventService scan
scheduling also had avoidable latency and queue pressure.

Supersedes #5749.

What is changed and how it works?

  • Move performance-mode to persisted changefeed ReplicaConfig, propagate
    it through dispatcher register/reset messages, and isolate EventStore
    subscriptions by mode.
  • Make dispatcher heartbeats, Maintainer wakeups, EventService scan
    continuation, and schema-blocked retry changefeed-scoped. The retry ticker
    starts lazily only after the first low-latency changefeed.
  • Defer active-scan lifecycle registration until a worker actually invokes
    the scanner, reducing false queue pressure for no-scan tasks.
  • Correct ticdc_owner_resolved_ts_lag to consistently report current PD
    wall-clock age of the coordinator's global minimum resolved timestamp, with
    immediate refresh after complete node-report rounds.
  • Preserve concurrent Maintainer status updates across heartbeat snapshots.

Check List

Tests

  • Unit test
  • Manual test (three captures, two concurrent 100k-table changefeeds, 200k
    isolated subscriptions, 20.245 MB/s shared traffic for 30 minutes; all
    5,203 samples normal/equal. Low-latency owner checkpoint mean/P95/P99:
    0.879/1.059/1.158s; throughput: 1.770/2.358/2.508s.)

Questions

Will it cause performance regression or break compatibility?

Throughput remains the default. Existing configs that omit the field keep
throughput behavior. Opposite modes intentionally do not share EventStore
subscriptions. The 30-minute mixed-mode run passed without status or count
drift; aggregate CPU/RSS was 32.556 cores/25.862 GiB.

Do you need to update user documentation, design documentation or monitoring documentation?

Yes. User documentation for the new changefeed option should be added before
the PR is marked ready. The existing metric name is unchanged; its semantics
are clarified and made consistent.

Release note

Add a changefeed-level performance mode that allows low-latency and throughput
changefeeds to coexist, and reduce resolved-ts propagation latency.

Keep the low-latency mode focused on setting the logpuller advance interval to zero. Remove the additional batched heap update path and its helper after the 10k-region E2E test showed comparable mean and p95 latency without it.
Replace the periodic schema-capped scan retry with applied SchemaStore notifications. Serialize dispatcher scan scheduling with a short-lock state machine and coalesce worker continuations without dropping queued work.
Keep no-DML/no-DDL resolved notifications out of the scan worker queue while preserving dispatcher scan ownership. Gate continuation and schema-blocked recovery on low-latency mode, and cover queue-full recovery with a dropped-task metric.
@ti-chi-bot

ti-chi-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Aug 3, 2026
@ti-chi-bot

ti-chi-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nongfushanquan for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a85cd703-8947-4196-81df-6603bad445b4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

*: optimize resolved-ts and checkpoint propagation latency

1 participant