konflux pipeline down

[cgwalters]

Does anyone know how I can get a new version of quay.io/centos-bootc/bootc-image-builder to be built? It seems there have been no builds in 11 days and I've fixed a few bugs upstream that are thus still affecting users. So far I've advised those users to just build a local container, but since this is a recurring theme (some users have set up their own pipelines to build that container ...) I'd thought I'd ask here.

Reportedly this may be https://issues.redhat.com/browse/KONFLUX-9346

[cgwalters]

Related: We should also discuss somewhere the future of the centos-bootc namespace and the production builds.

A tension here is that bootc-image-builder is in theory at least partially OS independent, totally unlike the bootc base images.

There's really three options:

• Build this container in each OS namespace i.e. introduce quay.io/fedora/bootc-image-builder and quay.io/centos/bootc-image-builder
• Have upstream builds in e.g. ghcr.io/osbuild (or quay.io/osbuild, whatever)
• Both of the above

[supakeen]

We used to have upstream builds (at ghcr.io/osbuild/bootc-image-builder) but I think we disabled them because we wanted this to live downstream; cc @ondrejbudai who might know the backstory there.

Personally I'd much prefer if these things live upstream instead of downstream :)

[supakeen]

A related bug that this is blocking is #968. There's also some workarounds for users linked there.

[cgwalters]

Personally I'd much prefer if these things live upstream instead of downstream :)

The centos-bootc one feels more "midstream" to me than upstream in that sense, but yes.

[achilleas-k]

• Have upstream builds in e.g. ghcr.io/osbuild (or quay.io/osbuild, whatever)

I'm very much in favour of this.

We used to have upstream builds (at ghcr.io/osbuild/bootc-image-builder) but I think we disabled them because we wanted this to live downstream

I don't remember the details either, but for me dropping the ghcr build has caused nothing but pain. If I'm testing the "latest" version of BIB to check if a bug is reproducible upstream, I always need to build it locally. Also, whenever someone reports a bug we think might be already fixed upstream, we have to ask them to build BIB main themselves to verify (or do it ourselves). Having a container that we know is built from upstream sources and gets updated a few mins after any PR is merged will save us a lot of trouble.

At this point, I don't even care what the reasons were for dropping the ghcr builds. Let's bring them back.

[cgwalters]

Yes. That said, Red Hat products will continue to be required to build with Konflux downstream.
One obvious thing to start with what you have in PR #1001 is it's only building for x86_64. I did some web searching which turns up praetorian-inc/noseyparker#214 which I think is a decent pattern that we could probably try to productize more and share across upstreams.

We may also end up wanting to maintain shared tools to keep Konflux pipelines and GHA in sync. (In general I've been trying to move load-bearing logic outside of shell-script-in-YAML in GHA in that vein)

[achilleas-k]

Yes. That said, Red Hat products will continue to be required to build with Konflux downstream.

Of course. Bringing back ghcr builds doesn't solve the main issue here (which is why I'm not closing this), but it gets rid of some of the more frustrating side effects, especially for non RH users.

One obvious thing to start with what you have in PR #1001 is it's only building for x86_64. I did some web searching which turns up praetorian-inc/noseyparker#214 which I think is a decent pattern that we could probably try to productize more and share across upstreams.

Thanks for pointing that out and for the link. I agree we need more work on this. My PR was a quick revert of the commit that disabled the builds to get something out quickly.

We may also end up wanting to maintain shared tools to keep Konflux pipelines and GHA in sync. (In general I've been trying to move load-bearing logic outside of shell-script-in-YAML in GHA in that vein)

That's a great idea!

[supakeen]

There still have been no builds produced by Konflux, shall we remove references to the downstream containers from our documentation until this is resolved?

[shi2wei3]

A new build is released now, since the konflux bug is not fixed yet, so manual intervention might be needed, so it's not stable.

[cgwalters]

Thanks @shi2wei3 !

On the overall topic there's definitely no issues AFAIK with ensuring that the team here has operational control and visibility over the current CentOS Konflux pipelines. Here I think at least ⅔ of the problem is that the release policies being deleted was quite painful to trace down and not obvious (as opposed to straight up build failures).

But if you want I'm also personally fine with just building the containers via some other mechanism and we can ensure that what's in quay.io/centos-bootc/bootc-image-builder syncs from there instead of being built via Konflux.

(That said of course, a lot of details here...like the signing key will necessarily change, if they remain signed at all)