CI: generic runtime tests by aparcar · Pull Request #13785 · openwrt/packages

aparcar · 2020-10-27T05:39:44Z

Requested via #13589

Most compiled programs offer a way to output the installed version, usually via arguments like `-v`, or `--version`. This fact can be used to runtime test compiled in a generic way. If the binary prints the compiled version it proves that the program loaded correctly on the specific architecture. While this obviously lacks testing of actual functionallity, it proves a package successfully compiled, installed and loaded related binaries. In cases where no package specific `test.sh` is found, try a variation of common way to print a programs version. Signed-off-by: Paul Spooren <[email protected]>

aparcar · 2020-10-27T19:23:20Z

@jefferyto Here we face the first problem. The vim package contains a tool called xxd which is an entirely different version:

Testing package xxd in version 8.2 from vim
Installing xxd (8.2-2) to root...
Configuring xxd.
Use generic test
Test executable /usr/bin/xxd

It seems to be something like this:

$ xxd -v
xxd V1.10 27oct98 by Juergen Weigert

Signed-off-by: Paul Spooren <[email protected]>

jefferyto · 2020-10-29T09:59:31Z

I don't want to comment too much on the immediate issue with xxd because I'm not crazy about this PR as it stands. I think doing everything inside the container is too inflexible and you have no access to the package Makefile.

I suggest something like this (it doesn't need to be all done in one PR), for each package/subpackage:

Checking the package Makefile, if the package has opted into tests (as mentioned in Runtime test ideas/enhancements #13589 (comment))
- If the package does not have a custom test script, output the default test script to something like custom_test.sh or custom_test_<subpackage-name>.sh
- (optional) If the package has a customized command line argument (instead of --version -version ...), output the default test script with the custom command line argument into custom_test.sh / custom_test_<subpackage-name>.sh (perhaps there are other variables in the default test script that can be customized in a similar manner)
- If the package has a custom test script, output it into custom_test.sh / custom_test_<subpackage-name>.sh
Inside the container
- If there is a custom_test.sh / custom_test_<subpackage-name>.sh, run it
- Otherwise if there is a test.sh, run that

This way the content of test.sh can even be completely defined inside the Makefile (similar to config options, they can be defined inside the Makefile or sourced from a separate file).

champtar · 2020-10-29T13:17:02Z

We expect contributor to be able to fix CI failure, so we need to be able to replicate what the CI does as easily as possible
=> make package/aaa/test would be really nice
And also CI should be green for working packages, it's nice to test more things but if we end up ignoring 90% of the times is it really better

aparcar · 2020-11-18T04:22:25Z

Sorry this slipped through my notifications.

everything inside the container is too inflexible

Please elaborate what parts are too inflexible.

you have no access to the package Makefile

The test.sh script is placed literally next to the Makefile and therefore accessible from the container. In fact the entire packages.git is mounted within the Docker container.

We could use Makefile define to define tests rather than having separate files, not sure if that's a big advantage.

=> make package/aaa/test would be really nice

How do you envision this to be implemented? Your host system is unlikely to support all these architectures, likely not even x86-musl. Then you need a working opkg installing dependencies. You can make it work, but that's plenty of work. One could automatically spin up a dozen Docker containers, but then again that's what the CI is for. My lazy approach would be that people just draft PRs and push until the CI is happy. I don't expect anyone to test their PRs for 10 archs.

BKPepe · 2020-12-29T21:13:24Z

.github/workflows/entrypoint.sh

+				echo "Generic test failed"
+				exit 1
+			else
+				echo "Test succesful or"


Shouldn't this be without or?

Suggested change

echo "Test succesful or"

echo "Test successful"

BKPepe · 2020-12-29T21:14:28Z

.github/workflows/entrypoint.sh

+				echo "Test succesful or"
+			fi
+		else
+			echo "No executeable found in package $PKG_NAME"


Suggested change

echo "No executeable found in package $PKG_NAME"

echo "No executable found in package $PKG_NAME"

BKPepe · 2020-12-29T21:18:34Z

Tested with lighttpd. Seems OK to me! More tested packages will follow soon.

aparcar · 2020-12-29T22:36:01Z

I'd like to get more input from @champtar and @jefferyto before continuing.

BKPepe · 2021-01-26T11:38:03Z

(optional) If the package has a customized command line argument (instead of --version -version ...), output the default test script with the custom command line argument into custom_test.sh / custom_test_<subpackage-name>.sh (perhaps there are other variables in the default test script that can be customized in a similar manner)

I agree on this one. We should have something like this.
There might be some packages, which does not have e.g. --version, -version and it that case check fails and in the logs, there is: generic test failed. In that case, there could be fallback to custom test or make it even complicated:

try to check for version
try to show help
try to run app without any parameters to check if it runs and that it does not segfault or does not show any errors
have custom script if method 1.,2. fails (they are not there).

Because I doubt that anyone wants to create his/her test.sh or ship the generic test.sh with each package folder, so having something by default 👍

TL;DR: I agree with @jefferyto's suggestions, but one difference that all packages will be using tests even though could be complicated for libraries.

BKPepe · 2025-05-20T07:09:09Z

It looks like several members are really depend on run testing packages inside CI/CD, but most of those packages does not have test.sh file at all and it is not run tested in the end. We should proceed with this PR.

Any thoughts, @1715173329 , @neheb, @robimarko ?

GeorgeSapkin · 2025-09-30T22:41:12Z

.github/workflows/entrypoint.sh

 	else
-		echo "No test.sh script available"
+		echo "Use generic test"
+	        PKG_FILES=$(opkg files "$PKG_NAME" | grep -E "/bin|/sbin|/usr/bin|/usr/sbin")


Need apk info -L support

GeorgeSapkin · 2025-09-30T22:41:57Z

.github/workflows/entrypoint.sh

+				for V in --version -version version -v -V; do
+					echo "Trying $FILE $V"
+					if "$FILE" "$V" 2>&1 | grep "$PKG_VERSION"; then
+						SUCCESS=1
+						break
+					fi
+				done


I think ensuring the main binary can display help should be enough for a success already, as a last resort after the version checks. This at least checks that binaries are valid and have the linked dependencies installed.

Also I think it makes sense to do grep -F if that's not the default.

I think ensuring the main binary can display help should be enough for a success already, as a last resort after the version checks. This at least checks that binaries are valid and have the linked dependencies installed

The problem is how? Because not all packages use exit 0 as a return. Maybe we can use grep to check the binary name itself?

I meant on top of --version -version version -v -V do something like --help, etc. Worst case scenario somebody's PR fails and they add a proper test.

Might be a good idea to also do an ldd (if it's a binary) and check if any linked dependencies are missing. And do this even if packages have their own tests.

wehagy

Also is better move to https://github.com/openwrt/actions-shared-workflows/blob/main/.github/dockerfiles_feeds/entrypoint.sh?

GeorgeSapkin · 2025-11-04T12:41:02Z

I ported this PR to openwrt/actions-shared-workflows#60 since this logic no-longer lives here. Can this be closed?

aparcar force-pushed the jefferyto-runtime branch 2 times, most recently from 4fd3b28 to df3a23f Compare October 27, 2020 18:54

aparcar force-pushed the jefferyto-runtime branch from df3a23f to 5366e19 Compare October 27, 2020 19:22

vim: remove generic test.sh

205f242

Signed-off-by: Paul Spooren <[email protected]>

BKPepe mentioned this pull request Nov 3, 2020

adblock: adapt openwrt rc.common changes #13849

Merged

BKPepe reviewed Dec 29, 2020

View reviewed changes

BKPepe mentioned this pull request Jul 5, 2023

net-mtools: Add new package to debug multicast setups #21496

Merged

BKPepe requested review from GeorgeSapkin and wehagy September 30, 2025 07:49

GeorgeSapkin reviewed Sep 30, 2025

View reviewed changes

GeorgeSapkin added the enhancement label Sep 30, 2025

wehagy reviewed Oct 2, 2025

View reviewed changes

GeorgeSapkin mentioned this pull request Nov 4, 2025

multi-arch-test-build: add generic package tests openwrt/actions-shared-workflows#60

Open

robimarko closed this Nov 4, 2025

	echo "No executeable found in package $PKG_NAME"
	echo "No executable found in package $PKG_NAME"

	PKG_FILES=$(opkg files "$PKG_NAME" \| grep -E "/bin\|/sbin\|/usr/bin\|/usr/sbin")
	PKG_FILES=$(opkg files "$PKG_NAME" \| grep -E "/bin\|/sbin\|/usr/bin\|/usr/sbin")

Conversation

aparcar commented Oct 27, 2020

Uh oh!

aparcar commented Oct 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jefferyto commented Oct 29, 2020

Uh oh!

champtar commented Oct 29, 2020

Uh oh!

aparcar commented Nov 18, 2020

Uh oh!

BKPepe Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

BKPepe Dec 29, 2020

Choose a reason for hiding this comment

Uh oh!

BKPepe commented Dec 29, 2020

Uh oh!

aparcar commented Dec 29, 2020

Uh oh!

BKPepe commented Jan 26, 2021

Uh oh!

BKPepe commented May 20, 2025

Uh oh!

GeorgeSapkin Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

wehagy Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeSapkin Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

wehagy Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeSapkin Oct 2, 2025

Choose a reason for hiding this comment

Uh oh!

wehagy left a comment

Choose a reason for hiding this comment

Uh oh!

GeorgeSapkin commented Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

aparcar commented Oct 27, 2020 •

edited

Loading