Field names must be unique. Found duplicates

[ulnit]

sink to kafka cause exception when alter table add column.
'IllegalArgumentException: Field names must be unique. Found duplicates ' .

[yuxiqian]

Hi @ulnit, could you please share the full log of the failed job with full exception trace?

[potterhe]

use cdc pipeline，mysql 2 doris。 code head hash(branch master): 5edd199

2025-03-28 16:25:39,394 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: Flink CDC Event Source: mysql -> Transform:Schema (1/1)#16 (2f8fb439e9530cb28bc301b427e73755_cbc357ccb763df2852fee8c4fc7d55f2_0_16) switched from RUNNING to FAILED with failure cause:
java.lang.IllegalArgumentException: Field names must be unique. Found duplicates: [xxx]
	at org.apache.flink.cdc.common.types.RowType.validateFields(RowType.java:158) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.common.types.RowType.<init>(RowType.java:54) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.common.types.RowType.of(RowType.java:183) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.common.types.RowType.of(RowType.java:175) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.runtime.typeutils.DataTypeConverter.toRowType(DataTypeConverter.java:81) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.runtime.operators.transform.PreTransformChangeInfo.of(PreTransformChangeInfo.java:110) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.runtime.operators.transform.PreTransformOperator.cacheChangeSchema(PreTransformOperator.java:361) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.runtime.operators.transform.PreTransformOperator.processElement(PreTransformOperator.java:294) ~[flink-cdc-dist-3.3.0.jar:3.3.0]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:310) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter$OutputCollector.collect(MySqlRecordEmitter.java:147) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at java.util.ArrayList.forEach(Unknown Source) ~[?:?]
	at org.apache.flink.cdc.debezium.event.DebeziumEventDeserializationSchema.deserialize(DebeziumEventDeserializationSchema.java:93) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitElement(MySqlRecordEmitter.java:120) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.processElement(MySqlRecordEmitter.java:96) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlPipelineRecordEmitter.processElement(MySqlPipelineRecordEmitter.java:116) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:73) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter.emitRecord(MySqlRecordEmitter.java:46) ~[flink-cdc-pipeline-connector-mysql-3.3.0.jar:3.3.0]
	at org.apache.flink.connector.base.source.reader.SourceReaderBase.pollNext(SourceReaderBase.java:203) ~[flink-connector-files-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:422) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:638) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:973) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:917) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:949) [flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [flink-dist-1.20.0.jar:1.20.0]
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist-1.20.0.jar:1.20.0]
	at java.lang.Thread.run(Unknown Source) [?:?]