cdc: support column selector for storage sink - #21862
Conversation
Signed-off-by: wk989898 <nhsmwk@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
missing-translation-status
📝 WalkthroughWalkthrough新增 TiCDC 云存储同步的列选择配置说明。文档覆盖表匹配、列名匹配、排除列、默认同步行为，以及主键或唯一键约束。 Changes列选择功能
Estimated code review effort: 1 (Trivial) | ~2 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: a34cb9c3-ba32-4134-9274-15c395c5aebe
📒 Files selected for processing (1)
ticdc/ticdc-sink-to-cloud-storage.md
| 以如下示例配置文件中的 `column-selectors` 配置项为例： | ||
|
|
||
| ```toml | ||
| [sink] | ||
| column-selectors = [ | ||
| {matcher = ['test.t1'], columns = ['a', 'b']}, | ||
| {matcher = ['test.*'], columns = ["*", "!b"]}, | ||
| {matcher = ['test1.t1'], columns = ['column*', '!column1']}, | ||
| {matcher = ['test3.t'], columns = ["column?", "!column1"]}, | ||
| ] | ||
| ``` |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
git ls-files | rg '(^|/)ticdc-sink-to-cloud-storage\.md$|column_selector\.go$|.*columnselector.*'
echo "== relevant doc snippet =="
if [ -f ticdc/ticdc-sink-to-cloud-storage.md ]; then
sed -n '300,350p' ticdc/ticdc-sink-to-cloud-storage.md | cat -n
fi
echo "== search column-selector docs =="
rg -n "column-selectors|column selector|规则按配置顺序匹配|第一条匹配|first match|priority|priority order" ticdc || true
echo "== locate source implementation =="
fd -a 'column_selector\.go$' . | sed 's#^\./##'
echo "== inspect source outline/grep if available =="
for f in $(fd 'column_selector\.go$' .); do
echo "--- $f ---"
wc -l "$f"
rg -n "func |type |first|order|matcher|ColumnSelector|Match" "$f" | head -120
doneRepository: pingcap/docs-cn
Length of output: 2848
🌐 Web query:
TiCDC column-selectors first matching rule matches table example test.t1 test.*
💡 Result:
In TiCDC, column-selectors follow a first-matching rule, meaning that for any given table, the system applies the configuration from the first rule in the list that matches that table [1][2]. For the provided example configuration: [sink] column-selectors = [ {matcher = ['test.t1'], columns = ['a', 'b']}, {matcher = ['test.'], columns = ["", "!b"]}, ] The behavior is as follows: 1. For table test.t1: It matches the first rule ({matcher = ['test.t1']}). Therefore, only columns 'a' and 'b' are replicated [1][2]. 2. For other tables in the test database (e.g., test.t2): They do not match the first rule, but they do match the second rule ({matcher = ['test.*']}). Consequently, all columns except 'b' are replicated [1][2]. If a table does not match any of the defined rules, TiCDC defaults to replicating all columns [1][2]. Note that this feature only takes effect when the downstream sink is Kafka [3][4][1].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/ticdc-sink-to-kafka/
- 2: https://github.com/pingcap/docs/blob/master/ticdc/ticdc-sink-to-kafka.md
- 3: https://docs.pingcap.com/tidb/v8.1/ticdc-changefeed-config/
- 4: https://docs.pingcap.com/tidb/stable/ticdc-changefeed-config/
补充 column-selectors 的匹配优先级。
test.t1 会同时匹配第 1 条和第 2 条规则。文档应说明：规则按配置顺序匹配；表匹配多条规则时，TiCDC 使用第一条匹配的规则。否则调整规则顺序会导致不同表的列集合发生变化。
可直接提交的修改
{matcher = ['test3.t'], columns = ["column?", "!column1"]},
]+规则按配置顺序匹配。表匹配多条规则时，TiCDC 使用第一条匹配的规则。
</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
以如下示例配置文件中的 `column-selectors` 配置项为例：
Source: Path instructions
| > **注意：** | ||
| > | ||
| > 经过 `column-selectors` 规则过滤后，表中的数据必须要有主键或者唯一键被同步，否则在 changefeed 创建或运行时会报错。 |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
fd -a 'ticdc-sink-to-cloud-storage\.md$' . || true
echo "== lines 320-355 =="
if [ -f ticdc/ticdc-sink-to-cloud-storage.md ]; then
sed -n '320,355p' ticdc/ticdc-sink-to-cloud-storage.md | cat -n
fi
echo "== search column selectors wording and related examples =="
rg -n "column-selectors|主键|唯一键|主键列|唯一键列|composite|composite.*key" ticdc/ticdc-sink-to-cloud-storage.md cdc/sink/dmlsink -g '*.go' 2>/dev/null | head -200 || true
echo "== inspect implementation around column selector =="
if [ -f cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go ]; then
wc -l cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go
ast-grep outline cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go --view compact 2>/dev/null || true
sed -n '1,260p' cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go
fi
echo "== search usages for returned errors / selection logic =="
rg -n "selected.*key|primary|unique|composite|ErrPrimary*|column selector|column-selector|Selector" cdc/sink/dmlsink/mq/transformer/columnselector -g '*.go' | head -200 || trueRepository: pingcap/docs-cn
Length of output: 2105
🌐 Web query:
tiflow column_selector composite primary key unique key Must select all columns error
💡 Result:
In TiCDC (part of the TiFlow project), if you encounter errors or unexpected behavior when using column-selectors with tables that have composite primary keys or unique keys, it is often due to the omission of essential columns required for identifying rows or maintaining consistency [1][2]. While there is no single "Must select all columns" error message that acts as a global rule for all operations, TiCDC requires that any columns necessary for the primary key or unique index—which the system uses as a handle to identify rows—must be present in the replicated data [2][3]. If column-selectors are configured to exclude these critical columns, TiCDC may be unable to properly process updates or deletes, potentially leading to errors, panics, or data inconsistency [1][4][5]. Key points to consider: 1. Requirement for Handle Columns: TiCDC uses primary key or unique index columns as "handle" columns to identify rows for DELETE and UPDATE operations [2][3]. If you use column-selectors to filter columns, ensure that the columns comprising your composite primary key or any unique keys are explicitly included [2]. 2. Troubleshooting Configuration: - Verify that your column-selectors configuration explicitly includes the necessary PK/UK columns [2]. - Be aware that certain configurations, such as delete-only-output-handle-key-columns = true, alter which columns are output; this parameter restricts output to only PK/UK columns [2][6]. 3. Known Issues: Recent versions of TiCDC have addressed various issues related to column-selectors and dispatchers, including panics when handling specific column configurations or virtual columns [1][4]. If you are experiencing a crash or "nil pointer" error, ensure you are using a patched version of TiCDC, as fixes for these issues have been merged into later releases [1][4]. 4. Validation: If the configuration fails during changefeed creation, check the TiCDC logs for ErrDispatcherFailed or similar errors, which often indicate that a specified column (such as a column in a composite key or a missing column) cannot be found in the table schema [7]. If you are seeing a specific error, it is recommended to review your column-selectors block in the changefeed configuration to ensure that the columns list covers all columns required by your downstream application and the primary/unique keys of the source table [2][8].
Citations:
- 1: panic on writeDebeziumFieldValue with column-selector and dispatcher tiflow#12208
- 2: https://docs.pingcap.com/tidb/v8.1/ticdc-changefeed-config/
- 3: https://docs.pingcap.com/tidb/v8.1/ticdc-manage-changefeed/
- 4: CDC panics when configuring the column dispatcher with virtual columns tiflow#12241
- 5: Always splitting update events if partition key changes tiflow#11297
- 6: https://docs.pingcap.com/tidb/v7.5/ticdc-changefeed-config/
- 7: cdc does not validate ineligibleTables for config of dispatcher tiflow#12253
- 8: https://docs.pingcap.com/tidb/v6.5/ticdc-open-api-v2/
🌐 Web query:
raw.githubusercontent.com pingcap tiflow master cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go
💡 Result:
The file cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go is part of the TiCDC component within the PingCAP TiFlow repository [1]. Its primary purpose is to implement column selection logic for data replication, allowing users to specify which columns to include or exclude when syncing data (typically to downstream systems like Kafka) [2][3]. This component works alongside the configuration field column-selectors in the TiCDC changefeed configuration [2][3]. By using this feature, users can define matchers for specific database tables and dictate precisely which columns are processed, supporting patterns and exclusions (e.g., using !column_name to exclude a column) [3]. Technical context: - Component: TiCDC (part of TiFlow) [1]. - Functionality: Enables granular control over data replication by filtering columns based on configuration rules [2][3]. - Path: cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go [4]. - Note: The TiCDC component in the pingcap/tiflow repository is maintained for versions up to v8.5.x; newer versions use a different architecture hosted at github.com/pingcap/ticdc [1].
Citations:
- 1: https://github.com/pingcap/tiflow
- 2: panic on writeDebeziumFieldValue with column-selector and dispatcher tiflow#12208
- 3: make the case-sensitive works on the whole dispatchers configuration tiflow#12103
- 4: The log prints too much when using CDC sync data to Kafka with nil values in column tiflow#11537
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== git remote / branch refs =="
git remote -v | sed -n '1,20p' || true
git branch -a | sed -n '1,40p' || true
git show-ref --heads --tags | sed -n '1,80p' || true
echo "== try direct raw file fetches =="
for url in \
"https://raw.githubusercontent.com/pingcap/tiflow/master/cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go" \
"https://raw.githubusercontent.com/pingcap/ticdc/master/cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go"
do
echo "--- $url ---"
curl -fsSLI "$url" | sed -n '1,20p' || echo "NOT FOUND"
if curl -fsSL "$url" > /tmp/column_selector.go; then
wc -l /tmp/column_selector.go
rg -n "New|ColumnSelector|Composite|Primary|unique|Error|selected|selector|Handle" /tmp/column_selector.go | head -120 || true
sed -n '1,260p' /tmp/column_selector.go | cat -n
fi
done
echo "== search local tiflow repo for relevant error selectors if available =="
if git rev-parse --show-toplevel >/dev/null 2>&1; then
find "$(git rev-parse --show-toplevel)" -type f -path '*columnselector*' -print 2>/dev/null | head -20 || true
find "$(git rev-parse --show-toplevel)" -type f -path '**/column_selector.go' -print 2>/dev/null | head -20 || true
find "$(git rev-parse --show-toplevel)" -type f -path '*dmlsink*' -name '*selector*.go' -print 2>/dev/null | head -40 || true
fiRepository: pingcap/docs-cn
Length of output: 13781
明确复合键必须完整保留。
column-selectors 的保留列必须包含主键的所有列；如果未保留全部主键列，则必须至少保留一个唯一键的所有列。部分保留复合主键或复合唯一键中的列仍会报错。
可直接提交的修改
-经过 `column-selectors` 规则过滤后，表中的数据必须要有主键或者唯一键被同步，否则在 changefeed 创建或运行时会报错。
+经过 `column-selectors` 规则过滤后，必须同步表的全部主键列，或至少一个唯一键的全部列；对于复合主键或复合唯一键，必须同步其中的所有列，否则在 changefeed 创建或运行时会报错。📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| > **注意：** | |
| > | |
| > 经过 `column-selectors` 规则过滤后，表中的数据必须要有主键或者唯一键被同步，否则在 changefeed 创建或运行时会报错。 | |
| > **注意：** | |
| > | |
| > 经过 `column-selectors` 规则过滤后，必须同步表的全部主键列，或至少一个唯一键的全部列；对于复合主键或复合唯一键，必须同步其中的所有列，否则在 changefeed 创建或运行时会报错。 |
Source: Path instructions
qiancai
added
the
do-not-merge/hold
|
|
||
| ## 列选择功能 | ||
|
|
||
| 自从 v8.5.8 开始，TiCDC 新架构支持列选择功能。可以对事件中的列进行选择，只将指定的列的数据变更事件发送到下游。 |
There was a problem hiding this comment.
TBD, currently no planned in v8.5.8 any more
qiancai
added
translation/doing
2 participants
First-time contributors' checklist
What is changed, added or deleted? (Required)
Add column selector description for ticdc storage sink
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions (in Chinese).
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit