Skip to content

Implement a proper fmf require: type: library support #3

@comps

Description

@comps

TMT/FMF do implement a full recursive metadata hierarchy, kind-of-hidden under the concept of "Beakerlib libraries".

I don't think we should hide it under Beakerlib and instead implement require: type: library in a generic fmf way (ie. export ATEX_SOMETHING as a path to the resolved tmpdir with all library requires the test specified).

The Beakerlib-specific part done in BeakerlibExecutor would then be just exporting BEAKERLIB_LIBRARY_PATH for rlImport.

The docs on https://tmt.readthedocs.io/en/stable/spec/tests.html#require are a bit unclear on the implementation, but, basically:

type: library
url: https://example.com/some/component
name: /path/in/repo

would clone https://example.com/some/component to some tmpdir into a dir named component (basename).

Then $tmpdir/component/path/in/repo is rsynced to the Remote, under ATEX_SOMETHING / component / path / in / repo (preserving the name path).

If nick is specified, it just replaces the auto-parsed basename:

type: library
url: https://example.com/some/component
name: /path/in/repo
nick: altname

would clone https://example.com/some/component to altname and rsync $tmpdir/altname/path/in/repo to ATEX_SOMETHING / altname / path / in / repo.


The cloning would probably be done via atex.executor.fmf.discover() or another library-resolving function, which would recursively download and parse fmf metadata of all the libraries, perhaps adding them to some libraries attribute in the FMFTests dataclass.

The key part is that the discovery needs a tmpdir for cloning all the url-referenced libraries, and FMFExecutor then needs access to that tmpdir to selectively configure (rsync) require for each individual test in run_test(), as different tests may have different nick for the same cloned repo, or different URLs that end in the same basename.
(Meaning FMFTests.libraries would index a bunch of mkdtemp() directories by URL, probably. Discovery would need a context manager to clean up the tmpdir.)

  • Alternatively, some resolve_libraries() could be called on a pre-existing FMFTests, to mkdtemp() the tmpdir for libraries. It itself could be the context manager for cleaning them up.

FMFExecutor could easily support non-url libraries by just symlinking the paths from self.work_dir / "tests" to ATEX_SOMETHING - cases like:

type: library
name: /certgen
nick: openssl

(ln -s /var/atex-some-workdir/tests/certgen ATEX_SOMETHING/openssl/certgen)

type: library
path: /tmp/library/openssl
name: /certgen

(ln -s /tmp/library/openssl/certgen ATEX_SOMETHING/openssl/certgen)


NOTE: tmt doesn't seem to handle basename conflicts on a per-test basis; it seems to just check if the destination exists and if it does, to ignore the require. So if two tests use different URLs that end in the same basename, the second test gets the first URL. If we replicate this, we wouldn't need per-test isolation using URL-indexed set of mkdtemp()s.

Also NOTE that tmt doesn't seem to adjust or otherwise resolve the library beyond require parsing - we could fix that by properly loading a full fmf Tree of each library repo, adjust it, find the requested path via Tree name (not on-disk path) and use the on-disk path from fmf Tree.sources (location of .fmf).
That would allow virtual library names sharing the same on-disk files, nicely aligning with https://fmf.readthedocs.io/en/stable/features.html#elasticity) .

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions