1.23.1 breaks provisioning beaker machines from tmt

[sharkcz]

Seems that commit 0cabc75 breaks provisioning Red Hat beaker systems from tmt. We are getting

/var/tmp/tmt/run-006

/plans/gating
    provision
        queued provision.provision task #1: default-0
        
        provision.provision task #1: default-0
        how: beaker
        arch: s390x
        image: rhel-9.5
        whiteboard: tmt-006-CvgeRHum
        fail: Failed to create Beaker job
    finish
        warn: Nothing to finish, no guests provisioned.

plan failed

The exception was caused by 1 earlier exceptions

Cause number 1:

    provision step failed

    The exception was caused by 1 earlier exceptions

    Cause number 1:

        Failed to create Beaker job

        The exception was caused by 1 earlier exceptions

        Cause number 1:

            ({'response': "\t<type 'exceptions.ValueError'>\n\tlinux is not a valid group"}, {'name': 'rhel-9.5-s390x', 'distro': 'RHEL-9.5%', 'os': 'rhel-9.5', 'job_group': 'linux', 'meta_distro': False, 'arch': 's390x', 'variant': 'BaseOS', 'job_owner': None, 'ks_meta': None, 'kernel_options': None, 'kernel_options_post': None, 'retention_tag': 'audit', 'product': '[internal]', 'whiteboard': 'tmt-006-CvgeRHum', 'priority': 'Normal', 'tasks': [{'name': '/distribution/dummy', 'role': 'STANDALONE'}], 'ks_append': [], 'hostRequires': None, 'distro_tags': [], 'host_id': 'rhel-9.5-s390x'})

            The exception was caused by 1 earlier exceptions

            Cause number 1:

                <Fault 1: "<type 'exceptions.ValueError'>:linux is not a valid group">

with

tmt-1.40.0-1.fc40.noarch
beaker-common-29.1-1.fc40.noarch
beaker-redhat-0.2.1-15.fc40eng.noarch
python3-mracklib-1.23.1-1.fc40.noarch
python3-mrack-beaker-1.23.1-1.fc40.noarch

[sharkcz]

downgrading to 1.23.0 "fixes" the issue

[skycastlelily]

This issue is because teemtee/tmt#3397 has not been merged yet due to the Christmas break and then the tmt test issue(still there), I already contact tmt lead, and hopefully that mr would be merged soon.

[sharkcz]

thanks, was wondering whether there isn't an update in tmt missing :-)

[psss]

The tmt part has been merged:

• Support creating Beaker jobs using beaker-job-group key for mrack teemtee/tmt#3397

Will be released in 1.41.

[dav-pascual]

I am reopening this issue because this is causing a regression when trying mrack >1.20 in our CI systems.
The issue was a missunderstanding between "group" concept in mrack and Beaker.
In mrack "group" refers to groups of hosts, and it's used to assign the right ansible group and as a common key across different providers. In beaker, groups refer to groups of users, totally different concepts, hence why "not a valid group" error.

I'll need to revert 0cabc75 changes. To support specifying beaker group we'll add an extra key (e.g. beaker-job-group) and update the mapping in tmt integration to avoid any collateral damage.

@psss FYI

[skycastlelily]

Please don't revert this, it will break tmt users,thanks for your understanding.

we could walk around your situation by adding the following to mrack:

diff --git a/src/mrack/transformers/beaker.py b/src/mrack/transformers/beaker.py
index ec561e0..0250541 100644
--- a/src/mrack/transformers/beaker.py
+++ b/src/mrack/transformers/beaker.py
@@ -200,11 +200,15 @@ class BeakerTransformer(Transformer):
     def create_host_requirement(self, host):
         """Create single input for Beaker provisioner."""
         distro, variant = self._get_distro_and_variant(host)
+        if host["whiteboard"].startswith("tmt"):
+            host["job_group"] = host.get("group") or host.get("beaker-job-group") 
         specs = {
             "name": host["name"],
             "distro": distro,
             "os": host["os"],
-            "job_group": host["group"],
+            "group": host["group"],
+            "job_group": host.get("job_group"),
             "meta_distro": "distro" in host,
             "arch": host.get("arch", "x86_64"),
             "variant": variant,

after a period of time,ie, when tmt users are all likely get the latest mrack, I'm gonna open a tmt mr to have it pass beaker-job-group to mrack instead of translate beaker-job-group to group as it does now. And then, we could remove

+        if host["whiteboard"].startswith("tmt"):
+            host["job_group"] = host.get("group") or host.get("beaker-job-group") 

change "job_group": host.get("job_group") -> "job_group": host.get("beaker_job_group"),

in that way, users won't feel any break.

[dav-pascual]

@skycastlelily AFAIK It doesn't have to break tmt if we update the mapping correctly in
https://github.com/teemtee/tmt/blob/main/tmt/steps/provision/mrack.py#L1202
https://github.com/teemtee/tmt/blob/main/tmt/steps/provision/mrack.py#L1460

API key "beaker_job_group" for tmt metadata would remain unchanged.

Please don't revert this, it will break tmt users,thanks for your understanding.

we could walk around your situation by adding the following to mrack:

diff --git a/src/mrack/transformers/beaker.py b/src/mrack/transformers/beaker.py
index ec561e0..0250541 100644
--- a/src/mrack/transformers/beaker.py
+++ b/src/mrack/transformers/beaker.py
@@ -200,11 +200,15 @@ class BeakerTransformer(Transformer):
     def create_host_requirement(self, host):
         """Create single input for Beaker provisioner."""
         distro, variant = self._get_distro_and_variant(host)
+        if host["whiteboard"].startswith("tmt"):
+            host["job_group"] = host.get("group") or host.get("beaker-job-group") 
         specs = {
             "name": host["name"],
             "distro": distro,
             "os": host["os"],
-            "job_group": host["group"],
+            "group": host["group"],
+            "job_group": host.get("job_group"),
             "meta_distro": "distro" in host,
             "arch": host.get("arch", "x86_64"),
             "variant": variant,

after a period of time,ie, when tmt users are all likely get the latest mrack, I'm gonna open a tmt mr to have it pass beaker-job-group to mrack instead of translate beaker-job-group to group as it does now. And then, we could remove

+        if host["whiteboard"].startswith("tmt"):
+            host["job_group"] = host.get("group") or host.get("beaker-job-group") 

change "job_group": host.get("job_group") -> "job_group": host.get("beaker_job_group"),

in that way, users won't feel any break.

Even if this worked, I'm not in favour of introducing mrack client specific patch here. It doesn't belong in this codebase.

[skycastlelily]

AFAIK It doesn't have to break tmt if we update the mapping correctly in

yeah, we need to update the mapping correctly, that's for sure, and that is my step2:" I'm gonna open a tmt mr to have it pass beaker-job-group to mrack instead of translate beaker-job-group to group as it does now"

But a likely senario would be : users update tmt to the latest version, which pass beaker-job-group to mrack, while using old mrack , which doesn't handle beaker-job-group,and that's the main break I mean.

[dav-pascual]

AFAIK It doesn't have to break tmt if we update the mapping correctly in

yeah, we need to update the mapping correctly, that's for sure, and that is my step2:" I'm gonna open a tmt mr to have it pass beaker-job-group to mrack instead of translate beaker-job-group to group as it does now"

But a likely senario would be : users update tmt to the latest version, which pass beaker-job-group to mrack, while using old mrack , which doesn't handle beaker-job-group,and that's the main break I mean.

Fair point. For that we can enforce a mrack version
e.g. https://github.com/teemtee/tmt/blob/main/pyproject.toml#L67

[skycastlelily]

yeah, we could, but we need to make mrack new version release a short time before tmt new version, if it's landed a long time before tmt new version.Tbo, I feel some tension to do it this way,and I see this NOt impossible scenario: users update mrack to latest version, but tmt with beaker-job-group is still not released yet.

 Even if this worked,  

Yeah, it will work

  I'm not in favour of introducing mrack client specific patch here. It doesn't belong in this codebase.

I know, and totally Make sense , it will likely be removed in 2 months :)