bats tests under docker ?

[rockandska]

Hi,

Will you be open to a PR who implement bats test under docker with multiple bash versions with/without gnu tools ?
The idea behind this, is to be able to :

• add eventually the same tests with fish shell (if bats is compatible) or other shell
• having a clean/closed test environment
• be able to source file tested to add additional tests
  But it implies to have docker on our dev env...

If yes, I could try to open a PR with minimal changes to have the same tests passing

[native-api]

Definitely worth trying out -- though while being careful about what it can actually achieve and what it cannot.

• add eventually the same tests with fish shell (if bats is compatible) or other shell
  • Different Bash versions -- that's definitely something that's not done now and is worth implementing. We did have bugs that only manifest in some new Bash versions.
  • Fish -- we already have some tests for Fish and in CI, we install Fish that's available in Ubuntu and Homebrew. Perhaps you want to test with some other versions as well?
• having a clean/closed test environment
  • Not sure what exactly that means. The tests are designed to be runnable in a live system (just like CPython's tests). We mock relevant parts of external/internal machinery to not depend on e.g. specific Python versions installed. Crucially, we also mock OS detection (an alternative would be to skip tests for a different OS -- then we will not be able to run all the tests locally). But we do not mock standard UNIX tools and we run tests in all relevant CI environments specifically to check how it would fare with OS specifics that are hard/impossible to predict.
  • I guess Docker would make it possible to test in other mainstream Linux distros without support for them from Github -- that would certainly be a big plus!
    • In particular, it would allow us to test installing build dependencies for different distros, even implement a maintained subcommand for that!
  • Not sure if it would allow us to test in more MacOS versions than what Github supports -- Docker probably can't emulate that
• be able to source file tested to add additional tests
  • not sure what this means
• But it implies to have docker on our dev env...
  • if we run the same tests, just inside a Docker container, that would not be a requirement. If you wish to run those locally, you can do the same.

[rockandska]

Definitely worth trying out -- though while being careful about what it can actually achieve and what it cannot.

That's the purpose of this discussion :)

Different Bash versions -- that's definitely something that's not done now and is worth implementing. We did have bugs that only manifest in some new Bash versions.

Should be doable without too much effort.
That's what I do in a personal project, for example :
https://github.com/rockandska/fzf-obc/actions/runs/17503962895
https://github.com/rockandska/fzf-obc/blob/v2/.github/workflows/pull_request.yml

Fish -- we already have some tests for Fish and in CI, we install Fish that's available in Ubuntu and Homebrew. Perhaps you want to test with some other versions as well?

Not the priority but could be achieved with some works

• having a clean/closed test environment
  • Not sure what exactly that means. The tests are designed to be runnable in a live system (just like CPython's tests). We mock relevant parts of external/internal machinery to not depend on e.g. specific Python versions installed. Crucially, we also mock OS detection (an alternative would be to skip tests for a different OS -- then we will not be able to run all the tests locally). But we do not mock standard UNIX tools and we run tests in all relevant CI environments specifically to check how it would fare with OS specifics that are hard/impossible to predict.

What I mean is to be able to source files directly with minimal mocking to be able to test bash function independently or part of libexec.
In my PR #3320, I had intention to add a bats test for the fix but was not able to do it, because I didn't find a way to source pyenv.d/exec/pip-rehash.bash in bats.

I guess Docker would make it possible to test in other mainstream Linux distros without support for them from Github -- that would certainly be a big plus!

• In particular, it would allow us to test installing build dependencies for different distros, even implement a maintained subcommand for that!

Yes, but the build part we'll be more cumbersome and could be address in a second PR

• Not sure if it would allow us to test in more MacOS versions than what Github supports -- Docker probably can't emulate that

I had eyes on https://github.com/sickcodes/Docker-OSX but didn't follow the achievement of the project

• But it implies to have docker on our dev env...
• if we run the same tests, just inside a Docker container, that would not be a requirement. If you wish to run those locally, you can do the same.

For bats tests, it is only a question of which shell you want to try, so the local test is not mandatory I think compared to the build test. Right ?

[rockandska]

285d58ddb110:/code# enable -f libexec/pyenv-realpath.dylib realpath 
285d58ddb110:/code# realpath /tmp/bats-run-BEnjoe/test/1/pyenv
285d58ddb110:/code# realpath /tmp/bats-run-BEnjoe/test/1/

In a container who's running alpine (musl) , the library realpath used do not return anything not even an error, so I plan to do not use it if an empty string is returned but should be addressed to at least return an error IMO.

[native-api]

That's a bug -- not clear yet whether in Pyenv or 3rd party.
Please report it separately and provide the necessary diagnostic information, then we'll be able to say more.

[rockandska]

I will try, but the bug is maybe/surely just related to the fact that the library was compiled on my computer (glibc) but used in the container with musl. (having it compiled on
What is the reason to use this library ? Is it used in internal by pyenv ? or just for the test ?

[native-api]

It's a helper to speed up cascade link resolution. If it's not available, a slower pure Bash implementation is used.

[rockandska]

So there is no incidence if it is disabled in docker test and a specific test suite for this library is done or should be done, right ?

[native-api]

Its failure to load is ignored unless the PYENV_NATIVE_EXT envvar is set. We currently set it in CI.
So if you don't set that envvar at your machine, the builtin's absense should not affect anything.

[rockandska]

FYI, I have a first release who achieve what we talked about here, but hit something odd with test-plugin on macos13 (not macos14).
When completed, I will open the PR and hope that it will correspond to what you have in mind.

[rockandska]

It is a shame that Bats do not have a "random" execution of tests in a file with a way to reproduce the same test, it will be simpler to show you how some of the actual tests are dependent of previous ones since they share some files.
Banging my head with a story of stub not executed when I isolate all the test, but now I understand ^^

In Macos-13, when isolated, the test yaml is linked from MacPorts fail because it reuses 'port-stub-run' from a previous test.
From what I've just seen, I should be able to force using port but will maybe need some help because it is hard to understand all the building logic

[rockandska]

Not sure if you're notified when a PR is open @native-api (I assume), so to be sure, I've opened one about the subject who covers other things too

[native-api]

I was notified of the PR. I've been busy with other stuff.

[rockandska]

Merged in master