Skip to content

cdc: remove 2.4.0 as the default kafka version keyword #17832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ticdc/ticdc-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ TiCDC 中的 Changefeed 和 Task 是两个逻辑概念,前者是分配同步
例如:

```
cdc cli changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1"
cdc cli changefeed create --server="http://127.0.0.1:8300" --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=${kafka_version}&partition-num=6&max-message-bytes=67108864&replication-factor=1"
cat changefeed.toml
......
[sink]
Expand Down
4 changes: 2 additions & 2 deletions ticdc/ticdc-canal-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Event 分为三类:
使用 `Canal-JSON` 时的配置样例如下所示:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-canal-json" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=canal-json"
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-canal-json" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}&protocol=canal-json"
```

## TiDB 扩展字段
Expand All @@ -33,7 +33,7 @@ Canal-JSON 协议本是为 MySQL 设计的,其中并不包含 TiDB 专有的 C
配置样例如下所示:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-canal-json-enable-tidb-extension" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=canal-json&enable-tidb-extension=true"
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-canal-json-enable-tidb-extension" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}&protocol=canal-json&enable-tidb-extension=true"
```

## Message 格式定义
Expand Down
2 changes: 1 addition & 1 deletion ticdc/ticdc-debezium.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ summary: 了解 TiCDC Debezium Protocol 的概念和使用方法。
使用 Debezium 消息格式时的配置样例如下所示:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-debezium" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&protocol=debezium"
cdc cli changefeed create --server=http://127.0.0.1:8300 --changefeed-id="kafka-debezium" --sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}&protocol=debezium"
```

Debezium 输出格式中包含当前行的 Schema 信息,以便下游消费者更好地理解当前行的数据结构。对于不需要输出 Schema 信息的场景,也可以通过在 changefeed 的配置文件或者 `sink-uri` 中将 `debezium-disable-schema` 参数设置为 `true` 来关闭 Schema 信息的输出。
Expand Down
2 changes: 1 addition & 1 deletion ticdc/ticdc-faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ TiCDC 为 service GC safepoint 设置的默认存活有效期为 24 小时,即
支持。注意:对于 Canal 协议,TiCDC 只支持 JSON 输出格式,对 protobuf 格式尚未提供官方支持。要开启 Canal 协议的输出,只需在 `--sink-uri` 配置中指定 `protocol` 为 `canal-json` 即可。例如:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=2.4.0&protocol=canal-json" --config changefeed.toml
cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri="kafka://127.0.0.1:9092/cdc-test?kafka-version=${kafka_version}&protocol=canal-json" --config changefeed.toml
```

> **注意:**
Expand Down
2 changes: 1 addition & 1 deletion ticdc/ticdc-simple-protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ TiCDC Simple Protocol 是一种行级别的数据变更通知协议,为监控
`sink-uri` 配置:

```shell
--sink-uri = "kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0"
--sink-uri = "kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}"
```

changefeed 配置:
Expand Down
12 changes: 6 additions & 6 deletions ticdc/ticdc-sink-to-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,14 @@ summary: 了解如何使用 TiCDC 将数据同步到 Kafka。
```shell
cdc cli changefeed create \
--server=http://10.0.10.25:8300 \
--sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=canal-json&kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1" \
--sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=canal-json&kafka-version=${kafka_version}&partition-num=6&max-message-bytes=67108864&replication-factor=1" \
--changefeed-id="simple-replication-task"
```

```shell
Create changefeed successfully!
ID: simple-replication-task
Info: {"sink-uri":"kafka://127.0.0.1:9092/topic-name?protocol=canal-json&kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1","opts":{},"create-time":"2023-11-28T22:04:08.103600025+08:00","start-ts":415241823337054209,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":".","config":{"case-sensitive":false,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"ddl-allow-list":null},"mounter":{"worker-num":16},"sink":{"dispatchers":null},"scheduler":{"type":"table-number","polling-time":-1}},"state":"normal","history":null,"error":null}
Info: {"sink-uri":"kafka://127.0.0.1:9092/topic-name?protocol=canal-json&kafka-version=${kafka_version}&partition-num=6&max-message-bytes=67108864&replication-factor=1","opts":{},"create-time":"2023-11-28T22:04:08.103600025+08:00","start-ts":415241823337054209,"target-ts":0,"admin-job-type":0,"sort-engine":"unified","sort-dir":".","config":{"case-sensitive":false,"filter":{"rules":["*.*"],"ignore-txn-start-ts":null,"ddl-allow-list":null},"mounter":{"worker-num":16},"sink":{"dispatchers":null},"scheduler":{"type":"table-number","polling-time":-1}},"state":"normal","history":null,"error":null}
```

- `--server`:TiCDC 集群中任意一个 TiCDC 服务器的地址。
Expand Down Expand Up @@ -63,7 +63,7 @@ Sink URI 用于指定 TiCDC 目标系统的连接信息,遵循以下格式:
一个通用的配置样例如下所示:

```shell
--sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=canal-json&kafka-version=2.4.0&partition-num=6&max-message-bytes=67108864&replication-factor=1"
--sink-uri="kafka://127.0.0.1:9092/topic-name?protocol=canal-json&kafka-version=${kafka_version}&partition-num=6&max-message-bytes=67108864&replication-factor=1"
```

URI 中可配置的的参数如下:
Expand Down Expand Up @@ -123,7 +123,7 @@ URI 中可配置的的参数如下:
- SASL/PLAIN

```shell
--sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&sasl-user=alice-user&sasl-password=alice-secret&sasl-mechanism=plain"
--sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}&sasl-user=alice-user&sasl-password=alice-secret&sasl-mechanism=plain"
```

- SASL/SCRAM
Expand All @@ -135,15 +135,15 @@ URI 中可配置的的参数如下:
SASL/GSSAPI `user` 类型认证:

```shell
--sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&sasl-mechanism=gssapi&sasl-gssapi-auth-type=user&sasl-gssapi-kerberos-config-path=/etc/krb5.conf&sasl-gssapi-service-name=kafka&sasl-gssapi-user=alice/for-kafka&sasl-gssapi-password=alice-secret&sasl-gssapi-realm=example.com"
--sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}&sasl-mechanism=gssapi&sasl-gssapi-auth-type=user&sasl-gssapi-kerberos-config-path=/etc/krb5.conf&sasl-gssapi-service-name=kafka&sasl-gssapi-user=alice/for-kafka&sasl-gssapi-password=alice-secret&sasl-gssapi-realm=example.com"
```

`sasl-gssapi-user` 和 `sasl-gssapi-realm` 的值与 kerberos 中指定的 [principle](https://web.mit.edu/kerberos/krb5-1.5/krb5-1.5.4/doc/krb5-user/What-is-a-Kerberos-Principal_003f.html) 有关。例如,principle 为 `alice/[email protected]`,则 `sasl-gssapi-user` 和 `sasl-gssapi-realm` 的值应该分别指定为 `alice/for-kafka` 和 `example.com`。

SASL/GSSAPI `keytab` 类型认证:

```shell
--sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=2.4.0&sasl-mechanism=gssapi&sasl-gssapi-auth-type=keytab&sasl-gssapi-kerberos-config-path=/etc/krb5.conf&sasl-gssapi-service-name=kafka&sasl-gssapi-user=alice/for-kafka&sasl-gssapi-keytab-path=/var/lib/secret/alice.key&sasl-gssapi-realm=example.com"
--sink-uri="kafka://127.0.0.1:9092/topic-name?kafka-version=${kafka_version}&sasl-mechanism=gssapi&sasl-gssapi-auth-type=keytab&sasl-gssapi-kerberos-config-path=/etc/krb5.conf&sasl-gssapi-service-name=kafka&sasl-gssapi-user=alice/for-kafka&sasl-gssapi-keytab-path=/var/lib/secret/alice.key&sasl-gssapi-realm=example.com"
```

SASL/GSSAPI 认证方式详见 [Configuring GSSAPI](https://docs.confluent.io/platform/current/kafka/authentication_sasl/authentication_sasl_gssapi.html)。
Expand Down
2 changes: 1 addition & 1 deletion ticdc/troubleshoot-ticdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,5 +135,5 @@ cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri="mysql://roo
该问题通常是由于 TiCDC 与 Kafka 集群连接失败导致。你可以通过检查 Kafka 的日志以及网络状况来排查。一个常见的原因是在创建同步任务时没有指定正确的 `kafka-version` 参数,导致 TiCDC 内部的 Kafka client 在访问 Kafka server 时使用了错误的 Kafka API 版本。你可以通过配置 [`--sink-uri`](/ticdc/ticdc-sink-to-kafka.md#sink-uri-配置-kafka) 指定正确的 `kafka-version` 参数来修复。例如:

```shell
cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri "kafka://127.0.0.1:9092/test?topic=test&protocol=open-protocol&kafka-version=2.4.0"
cdc cli changefeed create --server=http://127.0.0.1:8300 --sink-uri "kafka://127.0.0.1:9092/test?topic=test&protocol=open-protocol&kafka-version=${kafka_version}"
```