RHAIENG-2365, RHAIENG-2366, RHAIENG-2367: chore(build-args): update BASE_IMAGE references to use 3.2 AIPCC base images and add script for alignment automation

[jiridanek]

https://issues.redhat.com/browse/RHAIENG-2365
https://issues.redhat.com/browse/RHAIENG-2366
https://issues.redhat.com/browse/RHAIENG-2367

Description

Build failures analysis

Codeserver builds are still flaky as usual

Errors during downloading metadata for repository 'rhel-9-for-ppc64le-appstream-eus-rpms':
  - Curl error (91): SSL server certificate status verification FAILED for https://cdn.redhat.com/content/eus/rhel9/9.6/ppc64le/appstream/os/repodata/repomd.xml [No OCSP response received]
Error: Failed to download metadata for repo 'rhel-9-for-ppc64le-appstream-eus-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
subprocess exited with status 1

that is random network error

+ curl -sSfL https://download.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/t/texlive-tcolorbox-20200406-37.el9.noarch.rpm
+ rpm2cpio /dev/stdin
+ cpio -idmv
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
argument is not an RPM package
cpio: premature end of archive

another random network error in different build

unpacking failed (error: exit status 1; output: open /opt/app-root/lib/python3.12/site-packages/pandas/core/construction.py: no space left on device)
step-prepare-sboms :-

seen in https://konflux-ui.apps.stone-prod-p02.hjvn.p1.openshiftapps.com/ns/rhoai-tenant/applications/automation/pipelineruns/odh-workbench-jupyter-tensorflow-cuda-py312-on-pull-requesv9hmb/logs?task=build-images

  Error: Unable to find a match: unixODBC-devel

caused by missing ppc64le eus codeready repo, https://access.redhat.com/solutions/6977790 and https://redhat-internal.slack.com/archives/C01B8PCHP1V/p1765821273030669

this will be sorted out when our devops put their subscriptions in order in konflux

How Has This Been Tested?

• odh-workbench-jupyter-datascience-cpu-py312-on-pull-requesc66wg-3.2, Version: notebooks-d03af8c8720ee96f7feaba3973c162fc6d29a9cf-linux-x86-64

loads

• odh-workbench-jupyter-pytorch-rocm-py312-on-pull-request-xj77z-3.2, Version: notebooks-d03af8c8720ee96f7feaba3973c162fc6d29a9cf-linux-extra-fast-amd64

rocm with pytorch works

• quay.io/rhoai/pull-request-pipelines:notebooks-d03af8c8720ee96f7feaba3973c162fc6d29a9cf-linux-extra-fast-amd64@sha256:6320c9d2b94d422c07661d173ee6aee03cfa9bd499ab5195926631d68a20ba23, Version: notebooks-d03af8c8720ee96f7feaba3973c162fc6d29a9cf-linux-extra-fast-amd64

cuda with tensorflow works

Self checklist (all need to be checked):

• Ensure that you have run make test (gmake on macOS) before asking for review
• Changes to everything except Dockerfile.konflux files should be done in odh/notebooks and automatically synced to rhds/notebooks. For Konflux-specific changes, modify Dockerfile.konflux files directly in rhds/notebooks as these require special attention in the downstream repository and flow to the upcoming RHOAI release.

Merge criteria:

• The commits are squashed in a cohesive manner and have meaningful messages.
• Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
• The developer has manually tested the changes and verified that the changes work

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[jiridanek]

/build-konflux

[jiridanek]

/build-konflux

[openshift-ci]

@jiridanek: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/images	d03af8c	link	true	/test images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ysok

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ysok]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[atheo89]

/lgtm

For a future enhancement, it would be great to refactor scripts/update_konflux_args.py so the image tags aren't hardcoded inside the match/case block. If we pass them as arguments (e.g., using argparse), we can update the config files via CI/CD without modifying the Python script itself every time