The Quarkus Gradle compatibility nightmare and upgrade adventure - each and every time

[gwitsch]

So here's the story and my madness:
be aware of frustration and sarcasm

Once in a while you need to update dependencies. This time it's Quarkus.
Ok, so let's go, what could possibly go wrong? o.O
Famous last words

Well, we use LTS version for stability ;-)
I expect some minor changes no big issues.

Wait, what? The current version is 3.25.x and next LTS is 3.27.0 by the end of september. Usually we update quite in time. Fingers-crossed it will work. Let's go for the 3.25.x version and then keep on updating until LTS.
Just update the dependency declarations in the Gradle version catalog, run a gradle clean compileTestJava and wait for the magic.
Must have been black magic, Gradle does not even initialize the workspace.
Ok, so let's update Gradle first. Try by getting the latest version for Quarkus 3.8.3 first, just in case.
After trial and error this has been 8.12.1, so let's mark that for later

Quarkus	Gradle
3.8.3	8.12.1

Nice, now we at least now which version works at the moment.
Let's move on. Why does Gradle 8.14.3 not work?
Oh, Quarkus is using some internal Gradle API for their plugin.
What a genius move /s

Don't bother, we move on to the next Quarkus version - 3.25.0

Update dependency versions again and gradle clean compileTestJava another time.
And... no success, workspace cannot be initialized.
Ok, so get newer Gradle version. Let's try 8.14.3 this time.
And... oh it initializes the workspace, it compiles. Nice :-) Let's take a note:

Quarkus	Gradle
3.25.x	8.14.3

Well, not so fast. Run all applications using Quarkus and test them. They start, at least a good sign. Let's build a production version.
We first need to compile our inhouse logging extension for Quarkus. Great, it does not compile -.-
Let's check the release notes and migration guides. Oh "Changes in configuration for plugins, you need to migrate" ... whatever just change the code.
Some hours later, it compiles again. Let's give it another try. App compiles for production and starts. Let's run some actual tests and see if it still does what it's supposed to do. Looking good, let's check the log files.
Log files... log files? yes they are present. But wait... 0 kb in size, WTF o.O
Console log is fine, file log not. Ok, we added a custom log formatter with the extension. Maybe it's still broken.
After endless hours of debugging, thousands cups of coffee, I've found the issue. Someone thought it's a good idea to silently catch exceptions if formatting the log message fails in
org.jboss.logmanager.ExtHandler#publishToNestedHandlers
Wow, another genius move /s

Lol, it fails to load our utils class called within the log formatting method. Can't be possible, it's declared as a dependency in the extension's runtime project. Hmm, this extension need a weird project layout with deployment and runtime, I've put a note in the Gradle's build script for a workaround hack for dependencies. Maybe that does not work anymore. So, let's declare all dependencies in the deployment and runtime project. Whatever, it does not make sense, but... it has been the solution and log files are working again.
Nice, one problem solved and 3.25.0 is now working.
Let's go. Commit all changes, inform your colleagues it's done and everyone can try it.

10 mins later...
Colleague writes back, "just updated, cannot start my Quarkus app. Config in .env file is not found on my Windows machine"
Great, already been working on the update for weeks now and still not done nor working for everyone.
Let's try the next version(s), 3.25.1, 3.25.2 and so on.
Well, we had a temporary workaround, just put your config in application-dev.properties meanwhile and you're good to go.

But to be honest, i hate using libraries that are known to be buggy and a ticking timebomb. So what possibilities do we have?

• immediately update to next Quarkus versions on release and hope for fixes
• rollback and wait for LTS 3.27.0
• ignore issues it's just trunk development

We took option 1 and 3, updated every week and wasted a lot of time.
For a maintenance release, we also rolled back to 3.8.3 on a different branch, because a non-LTS version proofed, it's not usable for production!

3.27.0 release, finally it's here
Let's go. We fixed issues, it's been working with 3.25.3 and 4 already. Should not be difficult at all. And it's an LTS release.
Famous last words another time.

Update dependencies, recompile everything, start and test. All is looking good. Finally 🎉
Now, let's document and close that damn upgrade ticket finally.

Fast forward 2 week...

New Gradle release 9.1.0 came out.
We are at 8.14.3 still, maybe a good time to keep up to date and get the new version.
Famous last words again

Let's update, recompile. And...
Resolution of the configuration ':myExtension-deployment:runtimeClasspath' was attempted without an exclusive lock. This is unsafe and not allowed.
Great, another issue with Gradle and Quarkus. Let's take a note:

Quarkus	Gradle
3.27.0	8.14.3

Gradle 9.x.x does not work. But I am sure, there have been some fixes mentioned in the release notes. Maybe check again later.

Conclusion (TLDR)
Quarkus and Gradle are a nightmare. It's a lottery finding a working configuration. Don't even think about updating anything to a newer version used in Quarkus, especially not Gradle.
I don't know how others are handling these problem, but I am very interested in tips and tricks. Please share.

For the moment, I am done with Quarkus and looking for alternatives. It's a great framework, awesome documentation and examples, but the price you pay... reached the point where it's not worth the hassle and headache.
Frequent releases and updates are very welcome, but stability is the key point. And Quarkus disappointet me on every non-LTS upgrade. Breaking on stuff that you never might think of.

And have not even tried Java 25...

[gsmet]

Thanks for sharing! I won't go into all the specifics but I understand your pain. And tbh we are in a similar pain every time we update to a new version of Gradle in Quarkus. There are multiple ways to do a lot of things and it's hard to find the right way for a given version and even harder to find a way that will work in the next (and the next...).
We have been struggling a lot and, as you express it so vividly, it shows on the user side.

What doesn't help either is that there are a gazillion ways to set up your modules with Gradle, and then you have other plugins in the mix that can also do crazy things so it's extremely hard to cover all the flavors out there in our test suite.
We are improving with each bug report and addition to our test suite.

So, while I can understand your frustration, if you want things to improve (which apparently is not the case anymore but hey I have to try!), open issues with reproducers and we will be able to add more tests in our test suite and make sure we don't break things for you in the future.
There's really no other way.

[gwitsch]

Yes there will always be different ways to achieve some goals and it will be hard to test for most use cases.
But sticking to library or tools recommendations should be a building base for reliable tests.
E.g.: for Gradle there are lots of recommendations in this guide

Without checking all Quarkus' tests, it's hard to understand, why parallel execution is not tested well or the default. It's 2025, nearly any device containing a SoC has multicores. Especially devs and CI pipelines are trimmed to use maximum performance. So I doubt that anyone is still building their projects single threaded.

It's not a blame game! Maybe it grew over time and has not been considered? Or maybe it's already on some todo list? I don't have that insights, but status quo is frustrating.

From my experience there are three main areas that caused problems on every update I performed:

• parallel execution with org.gradle.parallel=true
• using internal Gradle API in Quarkus plugins. Searching for org.gradle.api.internal. in Quarkus source code gives me 14 relevant hits, all in devtools.
• ignored warnings from Gradle that would prevent problems when updating Gradle

[gsmet]

After endless hours of debugging, thousands cups of coffee, I've found the issue. Someone thought it's a good idea to silently catch exceptions if formatting the log message fails in org.jboss.logmanager.ExtHandler#publishToNestedHandlers

Could you clarify that?
I have checked the code and the exception is caught and then logged. Did you have a Throwable instead? I would be interested in knowing more about the error.

[gwitsch]

Quarkus was not able to load a class (SystemUtils) used in our custom Quarkus extension, which has been declared as an implementation dependency in runtime module only.
So I've received a NoClassDefFoundError swallowd in line 143 in org.jboss.logmanager.ExtHandler#publishToNestedHandlers.

The problem is weird, as it only happened locally in dev mode. A production build was working fine.
I resolved the problem by also adding the dependencies to the deployment module for the Quarkus extension as a runtimeOnly dependency. I also had to explicitly add the transitive dependency for the project containing the SystemUtils class.

The behaviour is confusing, as the same utils class has already been used during startup.
We have a log statement

LOGGER.info("available max heap size {}", () -> FileSizeFormatter.byteCountToDisplaySize(SystemUtils.getMaxHeapSize()));

which uses com.ith.share.shareutils.SystemUtils#getMaxHeapSize

And the logging extension failed by using com.ith.share.shareutils.SystemUtils#getProcessId.
Maybe this is related to the Gradle setup in use. Locally we use Gradle's dependency substitution to map dependencies to projects in the multi-module workspace.
Maybe, the extension might not be able to use that. Dependencies used in an extension always create a warning at startup locally:

> Task :webfhir:quarkusDev
15:18:06.347 WARN  io.quarkus.bootstrap.devmode.DependenciesFilter.getReloadableModules(DependenciesFilter.java:41) Live reload was disabled for the following project artifacts:
- com.ith.share:shareLogger:4.60.1.144
- com.ith.share:shareUtils:4.60.1.147