Konflux: switch to hermetic builds #4141
Conversation
|
Skipping CI for Draft Pull Request. |
|
This needs konflux-ci/build-definitions#2421 first |
jcapiitao
force-pushed
the
switch-to-hermetic
branch
from
2945ab3 to
7dccdd8
Compare
rpms.lock.yaml
Outdated
There was a problem hiding this comment.
Feels really weird to have specific mirrors hardcoded here, but I suspect those URLs are not actually used to fetch the packages but rather to record where it was cached from?
There was a problem hiding this comment.
Yeah, dunno why rpm-lockfile-prototype does not have a feature that takes the repo of the base container image for the libdnf resolution. Though, they have the feature that extracts installed packages from a container image see point 3. and determine only the necessary ones to fetch of the input file. We'll use this feature in this PR BTW, I'm doing my experimentation with jcapiitao#13 for now.
I suspect those URLs are not actually used to fetch the packages but rather to record where it was cached from?
Those URLs are actually used to do the libdnf resolution, and then generating the rpms.lock.yaml file. Then you can use a CLI tool that fetch those dependencies e.g https://github.com/hermetoproject/hermeto .
I'll update this PR with all the explanation very soon.
|
👍 from me. Locking cosa is something we've talked about for quite a while because it helps debugging issues. E.g. this would render coreos/coreos-ci-lib#165 obsolete. A little concerned about the PR churn, but let's see how it goes and we can adjust strategy as needed. |
jcapiitao
force-pushed
the
switch-to-hermetic
branch
from
7dccdd8 to
52c071f
Compare
|
/retest |
jcapiitao
force-pushed
the
switch-to-hermetic
branch
6 times, most recently
from
a0519f6 to
d8578f6
Compare
You mean the PRs submitted by MintMaker (i.e the Konflux Renovate service), or this PR ? Also, as a follow-up of this PR, I'll propose a github workflow to submit PRs automatically to update the lock YAML files. |
jcapiitao
force-pushed
the
switch-to-hermetic
branch
from
d8578f6 to
d508ed9
Compare
|
/retest |
That sounds good. I'm not sure if it's possible, though it'd be nice if it could auto-enable the auto-merge feature so it self-merges once CI passes. |
jcapiitao
force-pushed
the
switch-to-hermetic
branch
from
d508ed9 to
a29ab5f
Compare
This will enforce Konflux to prefetch the dependencies defined in the lock.yaml files with [1]. Then during the build, Konflux will 1. inject the repositories where the deps are stored, 2. configure the clients to pull the deps from there, 3. build without network. As rpm is still not fully supported [2], we have to enable `dev-package-managers` for now in the pipeline. All specific files enabling hermetic builds are located in the `ci/hermetic/` folder. You can find the helper scripts that automate the process of generating the lock YAML files, replacing the manual steps. The automation streamlines the workflow, reduces the chance of human error, and ensures consistency in the generated lock files. More details can be found in the updated README. This required adaptations to `build.sh` and the Dockerfile to support both hermetic and non-hermetic build processes. [1] https://github.com/konflux-ci/build-definitions/tree/main/task/prefetch-dependencies-oci-ta/0.2 [2] https://github.com/hermetoproject/hermeto?tab=readme-ov-file#package-managers
In 'rpms.in.yaml' file, we configured the rpm lockfile CLI tool to extract installed packages from the main Dockerfile which use the standard base image of Fedora. For consistency, we have to rbase all the container images of the project on the same image. Otherwise, we may end up with missing packages during the build (i.e tzdata is available in standard image, but not in minimal one, making the targetcli fail as tzdata was not add to the lock YAML file).
This is temporary while awaiting [1] to be merged and push in quay.io. [1] konflux-ci/build-definitions#2421
... because we need to have the CI/CD jobs ready to update automatically the lock files. Otherwise the Konflux pipelines will start failing because the Fedora RPMs defined in the lockfiles won't be available in remote repos a couple of days after (i.e Fedora update repos remove older RPMs).
jcapiitao
force-pushed
the
switch-to-hermetic
branch
from
a29ab5f to
137f8e7
Compare
|
/retest |
| @@ -0,0 +1,53 @@ | |||
| #!/bin/bash | |||
| set -euo pipefail | |||
There was a problem hiding this comment.
| set -euo pipefail | |
| set -euxo pipefail |
There was a problem hiding this comment.
As said in #4141 (comment)
I let the -x to the appreciation of the operator.
see commit messages