Skip to content

cdc: support column selector for storage sink #21862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wk989898 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

cdc: support column selector for storage sink #21862

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions ticdc/ticdc-sink-to-cloud-storage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,6 +312,32 @@ CDC000005.csv
}
```

## 列选择功能

自从 v8.5.8 开始，TiCDC 新架构支持列选择功能。可以对事件中的列进行选择，只将指定的列的数据变更事件发送到下游。

@qiancai qiancai Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TBD, currently no planned in v8.5.8 any more


以如下示例配置文件中的 `column-selectors` 配置项为例：

```toml
[sink]
column-selectors = [
{matcher = ['test.t1'], columns = ['a', 'b']},
{matcher = ['test.*'], columns = ["*", "!b"]},
{matcher = ['test1.t1'], columns = ['column*', '!column1']},
{matcher = ['test3.t'], columns = ["column?", "!column1"]},
]
```
Comment on lines +319 to +329

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file =="
git ls-files | rg '(^|/)ticdc-sink-to-cloud-storage\.md$|column_selector\.go$|.*columnselector.*'

echo "== relevant doc snippet =="
if [ -f ticdc/ticdc-sink-to-cloud-storage.md ]; then
  sed -n '300,350p' ticdc/ticdc-sink-to-cloud-storage.md | cat -n
fi

echo "== search column-selector docs =="
rg -n "column-selectors|column selector|规则按配置顺序匹配|第一条匹配|first match|priority|priority order" ticdc || true

echo "== locate source implementation =="
fd -a 'column_selector\.go$' . | sed 's#^\./##'

echo "== inspect source outline/grep if available =="
for f in $(fd 'column_selector\.go$' .); do
  echo "--- $f ---"
  wc -l "$f"
  rg -n "func |type |first|order|matcher|ColumnSelector|Match" "$f" | head -120
done

Repository: pingcap/docs-cn

Length of output: 2848

🌐 Web query:

TiCDC column-selectors first matching rule matches table example test.t1 test.*

💡 Result:

In TiCDC, column-selectors follow a first-matching rule, meaning that for any given table, the system applies the configuration from the first rule in the list that matches that table [1][2]. For the provided example configuration: [sink] column-selectors = [ {matcher = ['test.t1'], columns = ['a', 'b']}, {matcher = ['test.'], columns = ["", "!b"]}, ] The behavior is as follows: 1. For table test.t1: It matches the first rule ({matcher = ['test.t1']}). Therefore, only columns 'a' and 'b' are replicated [1][2]. 2. For other tables in the test database (e.g., test.t2): They do not match the first rule, but they do match the second rule ({matcher = ['test.*']}). Consequently, all columns except 'b' are replicated [1][2]. If a table does not match any of the defined rules, TiCDC defaults to replicating all columns [1][2]. Note that this feature only takes effect when the downstream sink is Kafka [3][4][1].

Citations:

补充 column-selectors 的匹配优先级。

test.t1 会同时匹配第 1 条和第 2 条规则。文档应说明：规则按配置顺序匹配；表匹配多条规则时，TiCDC 使用第一条匹配的规则。否则调整规则顺序会导致不同表的列集合发生变化。

可直接提交的修改 
     {matcher = ['test3.t'], columns = ["column?", "!column1"]},
 ]

+规则按配置顺序匹配。表匹配多条规则时，TiCDC 使用第一条匹配的规则。

</details>

<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

```suggestion
以如下示例配置文件中的 `column-selectors` 配置项为例：

Source: Path instructions


- 对于表 `test.t1`，只发送 `a` 和 `b` 两列的数据。
- 对于属于库 `test` 的表（除 `t1` 外），发送除 `b` 列之外的所有列的数据。
- 对于表 `test1.t1`，发送所有以 `column` 开头的列，但是不发送 `column1` 列的数据。
- 对于表 `test3.t`，发送所有以 `column` 开头且列名长度为 7 的列，但是不发送 `column1` 列的数据。
- 不匹配任何规则的表将不进行列过滤，发送所有列的数据。

> **注意：**
>
> 经过 `column-selectors` 规则过滤后，表中的数据必须要有主键或者唯一键被同步，否则在 changefeed 创建或运行时会报错。
Comment on lines +337 to +339

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file =="
fd -a 'ticdc-sink-to-cloud-storage\.md$' . || true

echo "== lines 320-355 =="
if [ -f ticdc/ticdc-sink-to-cloud-storage.md ]; then
  sed -n '320,355p' ticdc/ticdc-sink-to-cloud-storage.md | cat -n
fi

echo "== search column selectors wording and related examples =="
rg -n "column-selectors|主键|唯一键|主键列|唯一键列|composite|composite.*key" ticdc/ticdc-sink-to-cloud-storage.md cdc/sink/dmlsink -g '*.go' 2>/dev/null | head -200 || true

echo "== inspect implementation around column selector =="
if [ -f cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go ]; then
  wc -l cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go
  ast-grep outline cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go --view compact 2>/dev/null || true
  sed -n '1,260p' cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go
fi

echo "== search usages for returned errors / selection logic =="
rg -n "selected.*key|primary|unique|composite|ErrPrimary*|column selector|column-selector|Selector" cdc/sink/dmlsink/mq/transformer/columnselector -g '*.go' | head -200 || true

Repository: pingcap/docs-cn

Length of output: 2105

🌐 Web query:

tiflow column_selector composite primary key unique key Must select all columns error

💡 Result:

In TiCDC (part of the TiFlow project), if you encounter errors or unexpected behavior when using column-selectors with tables that have composite primary keys or unique keys, it is often due to the omission of essential columns required for identifying rows or maintaining consistency [1][2]. While there is no single "Must select all columns" error message that acts as a global rule for all operations, TiCDC requires that any columns necessary for the primary key or unique index—which the system uses as a handle to identify rows—must be present in the replicated data [2][3]. If column-selectors are configured to exclude these critical columns, TiCDC may be unable to properly process updates or deletes, potentially leading to errors, panics, or data inconsistency [1][4][5]. Key points to consider: 1. Requirement for Handle Columns: TiCDC uses primary key or unique index columns as "handle" columns to identify rows for DELETE and UPDATE operations [2][3]. If you use column-selectors to filter columns, ensure that the columns comprising your composite primary key or any unique keys are explicitly included [2]. 2. Troubleshooting Configuration: - Verify that your column-selectors configuration explicitly includes the necessary PK/UK columns [2]. - Be aware that certain configurations, such as delete-only-output-handle-key-columns = true, alter which columns are output; this parameter restricts output to only PK/UK columns [2][6]. 3. Known Issues: Recent versions of TiCDC have addressed various issues related to column-selectors and dispatchers, including panics when handling specific column configurations or virtual columns [1][4]. If you are experiencing a crash or "nil pointer" error, ensure you are using a patched version of TiCDC, as fixes for these issues have been merged into later releases [1][4]. 4. Validation: If the configuration fails during changefeed creation, check the TiCDC logs for ErrDispatcherFailed or similar errors, which often indicate that a specified column (such as a column in a composite key or a missing column) cannot be found in the table schema [7]. If you are seeing a specific error, it is recommended to review your column-selectors block in the changefeed configuration to ensure that the columns list covers all columns required by your downstream application and the primary/unique keys of the source table [2][8].

Citations:

🌐 Web query:

raw.githubusercontent.com pingcap tiflow master cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go

💡 Result:

The file cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go is part of the TiCDC component within the PingCAP TiFlow repository [1]. Its primary purpose is to implement column selection logic for data replication, allowing users to specify which columns to include or exclude when syncing data (typically to downstream systems like Kafka) [2][3]. This component works alongside the configuration field column-selectors in the TiCDC changefeed configuration [2][3]. By using this feature, users can define matchers for specific database tables and dictate precisely which columns are processed, supporting patterns and exclusions (e.g., using !column_name to exclude a column) [3]. Technical context: - Component: TiCDC (part of TiFlow) [1]. - Functionality: Enables granular control over data replication by filtering columns based on configuration rules [2][3]. - Path: cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go [4]. - Note: The TiCDC component in the pingcap/tiflow repository is maintained for versions up to v8.5.x; newer versions use a different architecture hosted at github.com/pingcap/ticdc [1].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== git remote / branch refs =="
git remote -v | sed -n '1,20p' || true
git branch -a | sed -n '1,40p' || true
git show-ref --heads --tags | sed -n '1,80p' || true

echo "== try direct raw file fetches =="
for url in \
  "https://raw.githubusercontent.com/pingcap/tiflow/master/cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go" \
  "https://raw.githubusercontent.com/pingcap/ticdc/master/cdc/sink/dmlsink/mq/transformer/columnselector/column_selector.go"
do
  echo "--- $url ---"
  curl -fsSLI "$url" | sed -n '1,20p' || echo "NOT FOUND"
  if curl -fsSL "$url" > /tmp/column_selector.go; then
    wc -l /tmp/column_selector.go
    rg -n "New|ColumnSelector|Composite|Primary|unique|Error|selected|selector|Handle" /tmp/column_selector.go | head -120 || true
    sed -n '1,260p' /tmp/column_selector.go | cat -n
  fi
done

echo "== search local tiflow repo for relevant error selectors if available =="
if git rev-parse --show-toplevel >/dev/null 2>&1; then
  find "$(git rev-parse --show-toplevel)" -type f -path '*columnselector*' -print 2>/dev/null | head -20 || true
  find "$(git rev-parse --show-toplevel)" -type f -path '**/column_selector.go' -print 2>/dev/null | head -20 || true
  find "$(git rev-parse --show-toplevel)" -type f -path '*dmlsink*' -name '*selector*.go' -print 2>/dev/null | head -40 || true
fi

Repository: pingcap/docs-cn

Length of output: 13781

明确复合键必须完整保留。

column-selectors 的保留列必须包含主键的所有列；如果未保留全部主键列，则必须至少保留一个唯一键的所有列。部分保留复合主键或复合唯一键中的列仍会报错。

可直接提交的修改 
-经过 `column-selectors` 规则过滤后，表中的数据必须要有主键或者唯一键被同步，否则在 changefeed 创建或运行时会报错。
+经过 `column-selectors` 规则过滤后，必须同步表的全部主键列，或至少一个唯一键的全部列；对于复合主键或复合唯一键，必须同步其中的所有列，否则在 changefeed 创建或运行时会报错。
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
> **注意：**
>
> 经过 `column-selectors` 规则过滤后，表中的数据必须要有主键或者唯一键被同步，否则在 changefeed 创建或运行时会报错。
> **注意：**
>
> 经过 `column-selectors` 规则过滤后，必须同步表的全部主键列，或至少一个唯一键的全部列；对于复合主键或复合唯一键，必须同步其中的所有列，否则在 changefeed 创建或运行时会报错。

Source: Path instructions


## 数据类型

本章节主要介绍 `schema_{table-version}_{hash}.json` 文件（以下简称为 schema 文件）中使用的各种数据类型。数据类型定义为 `T(M[, D])`，详见[数据类型概述](/data-type-overview.md#数据类型概述)。
Expand Down
Loading