TMT/FMF do implement a full recursive metadata hierarchy, kind-of-hidden under the concept of "Beakerlib libraries".
I don't think we should hide it under Beakerlib and instead implement require: type: library in a generic fmf way (ie. export ATEX_SOMETHING as a path to the resolved tmpdir with all library requires the test specified).
The Beakerlib-specific part done in BeakerlibExecutor would then be just exporting BEAKERLIB_LIBRARY_PATH for rlImport.
The docs on https://tmt.readthedocs.io/en/stable/spec/tests.html#require are a bit unclear on the implementation, but, basically:
type: library
url: https://example.com/some/component
name: /path/in/repo
would clone https://example.com/some/component to some tmpdir into a dir named component (basename).
Then $tmpdir/component/path/in/repo is rsynced to the Remote, under ATEX_SOMETHING / component / path / in / repo (preserving the name path).
If nick is specified, it just replaces the auto-parsed basename:
type: library
url: https://example.com/some/component
name: /path/in/repo
nick: altname
would clone https://example.com/some/component to altname and rsync $tmpdir/altname/path/in/repo to ATEX_SOMETHING / altname / path / in / repo.
The cloning would probably be done via atex.executor.fmf.discover() or another library-resolving function, which would recursively download and parse fmf metadata of all the libraries, perhaps adding them to some libraries attribute in the FMFTests dataclass.
The key part is that the discovery needs a tmpdir for cloning all the url-referenced libraries, and FMFExecutor then needs access to that tmpdir to selectively configure (rsync) require for each individual test in run_test(), as different tests may have different nick for the same cloned repo, or different URLs that end in the same basename.
(Meaning FMFTests.libraries would index a bunch of mkdtemp() directories by URL, probably. Discovery would need a context manager to clean up the tmpdir.)
- Alternatively, some
resolve_libraries() could be called on a pre-existing FMFTests, to mkdtemp() the tmpdir for libraries. It itself could be the context manager for cleaning them up.
FMFExecutor could easily support non-url libraries by just symlinking the paths from self.work_dir / "tests" to ATEX_SOMETHING - cases like:
type: library
name: /certgen
nick: openssl
(ln -s /var/atex-some-workdir/tests/certgen ATEX_SOMETHING/openssl/certgen)
type: library
path: /tmp/library/openssl
name: /certgen
(ln -s /tmp/library/openssl/certgen ATEX_SOMETHING/openssl/certgen)
NOTE: tmt doesn't seem to handle basename conflicts on a per-test basis; it seems to just check if the destination exists and if it does, to ignore the require. So if two tests use different URLs that end in the same basename, the second test gets the first URL. If we replicate this, we wouldn't need per-test isolation using URL-indexed set of mkdtemp()s.
Also NOTE that tmt doesn't seem to adjust or otherwise resolve the library beyond require parsing - we could fix that by properly loading a full fmf Tree of each library repo, adjust it, find the requested path via Tree name (not on-disk path) and use the on-disk path from fmf Tree.sources (location of .fmf).
That would allow virtual library names sharing the same on-disk files, nicely aligning with https://fmf.readthedocs.io/en/stable/features.html#elasticity) .
TMT/FMF do implement a full recursive metadata hierarchy, kind-of-hidden under the concept of "Beakerlib libraries".
I don't think we should hide it under Beakerlib and instead implement
require: type: libraryin a generic fmf way (ie. exportATEX_SOMETHINGas a path to the resolved tmpdir with all library requires the test specified).The Beakerlib-specific part done in BeakerlibExecutor would then be just exporting
BEAKERLIB_LIBRARY_PATHforrlImport.The docs on https://tmt.readthedocs.io/en/stable/spec/tests.html#require are a bit unclear on the implementation, but, basically:
would clone
https://example.com/some/componentto some tmpdir into a dir namedcomponent(basename).Then
$tmpdir/component/path/in/repois rsynced to the Remote, underATEX_SOMETHING / component / path / in / repo(preserving thenamepath).If
nickis specified, it just replaces the auto-parsed basename:would clone
https://example.com/some/componenttoaltnameand rsync$tmpdir/altname/path/in/repotoATEX_SOMETHING / altname / path / in / repo.The cloning would probably be done via
atex.executor.fmf.discover()or another library-resolving function, which would recursively download and parse fmf metadata of all the libraries, perhaps adding them to somelibrariesattribute in theFMFTestsdataclass.The key part is that the discovery needs a tmpdir for cloning all the url-referenced libraries, and FMFExecutor then needs access to that tmpdir to selectively configure (rsync)
requirefor each individual test inrun_test(), as different tests may have differentnickfor the same cloned repo, or different URLs that end in the same basename.(Meaning
FMFTests.librarieswould index a bunch ofmkdtemp()directories by URL, probably. Discovery would need a context manager to clean up the tmpdir.)resolve_libraries()could be called on a pre-existing FMFTests, tomkdtemp()the tmpdir for libraries. It itself could be the context manager for cleaning them up.FMFExecutor could easily support non-url libraries by just symlinking the paths from
self.work_dir / "tests"toATEX_SOMETHING- cases like:(
ln -s /var/atex-some-workdir/tests/certgen ATEX_SOMETHING/openssl/certgen)(
ln -s /tmp/library/openssl/certgen ATEX_SOMETHING/openssl/certgen)NOTE: tmt doesn't seem to handle basename conflicts on a per-test basis; it seems to just check if the destination exists and if it does, to ignore the require. So if two tests use different URLs that end in the same basename, the second test gets the first URL. If we replicate this, we wouldn't need per-test isolation using URL-indexed set of
mkdtemp()s.Also NOTE that tmt doesn't seem to
adjustor otherwise resolve the library beyondrequireparsing - we could fix that by properly loading a full fmfTreeof each library repo,adjustit, find the requested path viaTreename (not on-disk path) and use the on-disk path from fmfTree.sources(location of.fmf).That would allow virtual library names sharing the same on-disk files, nicely aligning with https://fmf.readthedocs.io/en/stable/features.html#elasticity) .