debezium/dbz#2194 Debezium Bridge Extension

[kmos]

Context

Today, the Quarkus Extensions provide both the Debezium Engine and the connector runtime, but they do not offer a built-in sink implementation. As a result, users must provide their own event consumer by implementing a method annotated with @Capturing.

Proposal

We would like to introduce an extension that bridges the Debezium Server sink infrastructure and exposes it as a sink target for the Quarkus Extensions.

With this approach, a Quarkus application or even a JBang script could act as a lightweight Debezium Server, leveraging the existing Debezium Server sink implementations without requiring users to write custom @Capturing handlers.

The following example illustrates the concept from both a code and dependency perspective:

//usr/bin/env jbang "$0" "$@" ; exit $?
//DEPS io.quarkus.platform:quarkus-bom:3.28.4@pom
//DEPS io.quarkus:quarkus-arc
//DEPS io.debezium.quarkus:debezium-quarkus-postgres:3.7.0-SNAPSHOT
//DEPS org.apache.kafka:kafka-clients:4.2.0
//DEPS io.debezium:debezium-server-redis:3.7.0-SNAPSHOT
//DEPS io.debezium.quarkus:debezium-quarkus-bridge:3.7.0-SNAPSHOT
//FILES application.properties
//JAVAC_OPTIONS -parameters
//JAVA_OPTIONS -Djava.util.logging.manager=org.jboss.logmanager.LogManager


@ApplicationScoped
public class StringConverter {

    @CustomConverter
    public ConverterDefinition<SchemaBuilder> bind(ConvertedField field) {
        return new ConverterDefinition<>(SchemaBuilder.string(), String::valueOf);
    }
}

The code executes CDC from postgres to kafka converting all the fields to String

closes debezium/dbz#2194

[Naros]

Should we also guard against the size being 0 to avoid IndexOutOfBoundsException below?

[kmos]

super good catch!

[mfvitale]

@kmos General stupid question. Should the bridge-extension be in the quarkus-extension repo?

Sorry I unintentionally clicked on close

[kmos]

@kmos General stupid question. Should the bridge-extension be in the quarkus-extension repo?

Sorry I unintentionally clicked on close

@mfvitale Well, it should be, but currently it isn't feasible. To move the bridge extension into the Quarkus repository, we would need to build debezium-server-api before the Quarkus extension to avoid a circular dependency. This would likely require moving debezium-server-api into the core repository, or adopting a similar approach.

[Naros]

I do think that's the right long-term goal, @kmos, just from a pure logistical point of view:

I assume your PoV is something like this:

• core repo: engine event contracts + sink contracts (e.g. Capturing, CapturingEvents, BatchEvent). We already have both a debezium-api and debezium-sink-api module there, so I think that's likely the best place, since those are plain interfaces/annotations.
• debezium-quarkus: this would be the extensions + the bridge
• debezium-server: sinks implement the debezium-sink-api contracts

The alternative would be to introduce a debezium-api repository that houses all the API contracts used across the portfolio, and then all downstream repos can consume from it.

Just moving the bits to the main repo gets us all we need without lots of interactions between the repos, which I think suffices for now, curious what others think.

[kmos]

I do think that's the right long-term goal, @kmos, just from a pure logistical point of view:

I assume your PoV is something like this:

• core repo: engine event contracts + sink contracts (e.g. Capturing, CapturingEvents, BatchEvent). We already have both a debezium-api and debezium-sink-api module there, so I think that's likely the best place, since those are plain interfaces/annotations.
• debezium-quarkus: this would be the extensions + the bridge
• debezium-server: sinks implement the debezium-sink-api contracts

The alternative would be to introduce a debezium-api repository that houses all the API contracts used across the portfolio, and then all downstream repos can consume from it.

Just moving the bits to the main repo gets us all we need without lots of interactions between the repos, which I think suffices for now, curious what others think.

exactly. You are on the point. What I didn't think it's the possibility to have another repository.

We should discuss how we want to manage all the APIs but I perfectly agree on the proposed solution (I am not sure about the repository).

[jpechane]

@kmos Could you please re-do the PR to use debezium-server-bom instead of injecting the dependency versions?