prepare/artifact: extend `verify` to user-defined `prepare/install` phases ordered before the verify phase

[vaibhavdaren]

Background

The prepare/artifact plugin supports a verify: true option that injects a verification phase (PrepareVerifyInstallation) to check that packages required by tests were actually installed from the configured artifact providers.

Currently, verification only covers the install phases injected by tmt itself (essential-requires, requires, recommends). User-defined prepare/install phases are excluded. This was a deliberate conservative choice to avoid imposing unexpected behavior on users who didn't ask for it.

The gap

A user can specify test dependencies either via require:/recommend: keys or via an explicit prepare/install phase — both achieve the same result. With the current implementation, the two approaches have diverging verification behavior: tmt-injected phases are verified, user-defined ones are not. This is a silent difference that may confuse users who expect consistent behavior.

Proposed approach

Extend verification to user-defined prepare/install phases, but gated on phase ordering: only include a phase if its order is less than the verify phase's order (PHASE_ORDER_PREPARE_VERIFY_INSTALLATION = 79).

This gives users a natural opt-out: set order: 80 (or higher) on a prepare/install phase to have it run after verification and be excluded from it. It also matches the general tmt convention that phase ordering controls execution semantics.

Phase order reference

Phase	Order
essential-requires (tmt-injected)	30
prepare/install default	50
requires (tmt-injected)	70
recommends (tmt-injected)	75
verify-installation	79

With this change, a user-defined prepare/install at the default order of 50 would be included in verification automatically. To opt out, the user sets order: 80+.

Open question

Is the opt-out mechanism (order: 80+) discoverable enough, or do we need an explicit verify: false flag on prepare/install to let users opt out without changing ordering semantics?

Related

• Auto-verify artifact packages against require/recommend #4726 (comment)

[LecrisUT]

Is the opt-out mechanism (order: 80+) discoverable enough, or do we need an explicit verify: false flag on prepare/install to let users opt out without changing ordering semantics?

My vote is to have both and probably more. as we find usages.

User-defined prepare/install phases are excluded. This was a deliberate conservative choice to avoid imposing unexpected behavior on users who didn't ask for it.

Regarding imposing unexpected behavior, because that would be the case for require, recommend, it's just that now we are adding an additional layer of complexity that the user has to be aware and documented about. My main vote in favor of this approach is because it would simplify the behavior and documentation burden.

From the Fedora-CI there are two approaches:

• Require all dependencies to be defined in the tests as part of the change proposal: estimated impact >225 packages, but they also overlap with the packages not being installed by default
• Expand the scope to include all prepare/install

On the other hand, the opt-out for the other users can be:

• Add the artifacts without verification (dangerous imo since it becomes an undefined behavior). Configurable from testing-farm or user plan
• Add the artifacts with disabled repo (preferred imo since that closer matches what the user actually intended). Configurable from testing-farm or user plan
• Adjust the prepare/install step to include verify: false or higher order. Configurable on the user plan side

---

I would mention that we should decide on this before the changes from multipipeline become live because after that this will become a breaking change and we do not have a mechanism to do backwards compatible changes. My guess is that @kkaarreell is the main user that is affected by these and should chime in if any of the opt-out would be employed.

[LecrisUT]

From the hacking session:

• We have a period between the enabling the multi-host-pipeline and making the announcement. During that time we would be able to check the actual impact of this change
• In order to actually pick up breakages we would not see it unless we also cover prepare/install. We would be able to revert quickly as needed during the testing period above. This would be able to be done on testing-farm side with either prepare --how install --update --verify or with policy
  • We need to add a --verify key to prepare/install
• After the testing period we are then locking in that either behavior is the golden path and document it as such