fix: bump kafka-clients version to 4.0.0

[monosoul]

This is a draft attempt at updating kafka-clients to the latest version (4.0.0).

Fixes #420

[Sage-Pierce]

@monosoul I was just looking into making a couple libraries compatible with Kafka 4.0.0 and ran into the issue you're addressing here. I wonder if you'd be amenable to reducing this changeset down to just the changes you've made in ReceiverRecord and DefaultTransactionManager, and defer upgrading this library to kafka-clients 4.0.0 (in favor of the latest 3.x release), such as to not yet force users to upgrade, as I'm finding it's a pretty non-trivial one. I think it could be less disruptive to users if this library was simply made compatible with 4.x, without forcing the transitive dependency, and I believe the changes you've made to migrate to non-deprecated APIs could accomplish that. I think the maintainers might be more apt to merge this quickly if it's just that. WDYT?

[monosoul]

@Sage-Pierce tbh, I think the effort required to make that library compatible with both Kafka clients version <4.0.0 and >=4.0.0 doesn't worth it. I'd just release it with a major version bump as a breaking change and that's it. Whoever need to use it with Kafka client <4.0.0 could simply use the old version.

[monosoul]

@violetagg any chance you could take a look here and maybe share your thoughts? Thank you!

[violetagg]

Please note this https://spring.io/blog/2025/05/20/reactor-kafka-discontinued

[monosoul]

Welp, that's a bummer. What's the recommended alternative?

[Sage-Pierce]

@monosoul - given @violetagg's update, and the remaining commitment to release "necessary fixes", would it sound more acceptable now to only fix the deprecated method/constructor usages? I've done some testing locally, and the changes you've made to ReceiverRecord and DefaultTransactionManager seem to be the only necessary changes to make this project at least compatible with 4.x. That would at least buy me/us some time to plan for an alternative.

@violetagg would only fixing these deprecated method/constructor usages be acceptable as a fix/patch release?

[jkonicki]

@Sage-Pierce Thank for the suggestion, we need to look into it as we would like to leave the project for the community in a good state. As mentioned in the blog, we will not be releasing a new minor or major, but if we can make a change in a maintenance release that does not break current compatibility we can consider doing so. The team will need some time to prioritize looking into this option and see what we can do.

As noted in the blog comments, it would also be an option to fork the reactor kafka repo and make the changes you need independently.

If there is another project maintained by the community, the team can review it and consider making that the recommend path for our current Reactor Kafka users.

[Sage-Pierce]

FYI @jkonicki - I took your recommendation to heart and looked into forking reactor-kafka to maintain/evolve it. Upon doing so, I discovered some opportunities for implementation simplification, behavioral/API enhancement, and performance improvement, but some of the things I wanted to change would result in slightly different behaviors and some backward incompatibilities.

I ended up deciding to tackle reimplementing the vast majority of reactor-kafka functionalities in another OSS project, and released for usage this week. The linked PR describes some of the enhancements+improvements that were made. Some example usages of it are provided in that project's wiki.

I would welcome your team's review of these resources, with the possibility of vetting them for recommendation as an alternative for current Reactor Kafka users. I'd certainly value any feedback!

cc: @monosoul @violetagg

[jkonicki]

Thanks for letting us know @Sage-Pierce !
cc: @chemicL @artembilan @sobychacko