Track runtime of make test on bencher.dev

[arielshaqed]

You can see results (1 measurement for now) here.

[nopcoder]

Very cool

1. if we trust running the tool I don't see an issue calling the action that set it up or the curl install instead of pulling manully from github.
2. can we use 'bencher run' without the file to and and report 'make test' step - the current code populate the github token in the same env (if we worried about passing it to untrusted process)

[nopcoder]

Just to keep the same style

Suggested change

	BENCHER_API_TOKEN: ${{secrets.BENCHER_API_TOKEN}}
	BENCHER_API_TOKEN: ${{ secrets.BENCHER_API_TOKEN }}

[arielshaqed]

Thanks!

[arielshaqed]

Thanks! Trying out the action; PTAL.

[arielshaqed]

Thanks!

[nopcoder]

Just making sure my comments were seen:

• if we trust running the tool I don't see an issue calling the action that set it up or the curl install instead of pulling manully from github.
• can we use 'bencher run' without the file to and and report 'make test' step - the current code populate the github token in the same env (if we worried about passing it to untrusted process)

the code looks the same as the initial review.

[AliRamberg]

I like this tool already! Maybe we need to measure it differently though, maybe more specific than the real/sys/user measures of the whole tests?

After reading some pages from the docs, there are really cool ways to integrate it in our CI
For tests there's actually an integration with go test -bench.
Not sure if it times the whole testing suite or each test independently but worth checking.

There're also build time and file size metrics we could collect. Probably around the goreleaser workflow...

[arielshaqed]

@nopcoder seen and accepted :-). Not sure why you saw nothing, but verified that everything is now visible on the PR.

@AliRamberg absolutely we can collect more statistics. Let's start with these, please! I want to start collecting data, and add more as we go along.

[nopcoder]

would prefer to reduce the complexity of the go test run just because we like to report the take under test-go-runtime.
not blocking, but if we plan to keep it for long time - or apply it in more places - we should switch to use '--build-time' with json adapter to capture the time it takes to 'make test' to compelte

[arielshaqed]

I tried to use this but we end up measuring how long it takes to download all the modules:

go list -f '{{.Dir}}/...' -m | xargs /opt/hostedtoolcache/go/1.24.10/x64/bin/go test -count=1 -coverprofile=cover.out -race -cover -failfast -parallel="4" ./...
go: downloading github.com/jamiealquiza/tachymeter v2.0.0+incompatible
go: downloading github.com/go-co-op/gocron v1.35.2
go: downloading github.com/jedib0t/go-pretty/v6 v6.6.7
go: downloading github.com/schollz/progressbar/v3 v3.13.1
go: downloading github.com/manifoldco/promptui v0.9.0
go: downloading github.com/stretchr/testify v1.11.1
[...]

I will split the test-go target into test-go-build and test-go-run, so we can measure only the second. Obviously CI users care about both, but equally obviously we care mostly about time to run tests.

[arielshaqed]

I tried to use this but we end up measuring how long it takes to download all the modules:

go list -f '{{.Dir}}/...' -m | xargs /opt/hostedtoolcache/go/1.24.10/x64/bin/go test -count=1 -coverprofile=cover.out -race -cover -failfast -parallel="4" ./...
go: downloading github.com/jamiealquiza/tachymeter v2.0.0+incompatible
go: downloading github.com/go-co-op/gocron v1.35.2
go: downloading github.com/jedib0t/go-pretty/v6 v6.6.7
go: downloading github.com/schollz/progressbar/v3 v3.13.1
go: downloading github.com/manifoldco/promptui v0.9.0
go: downloading github.com/stretchr/testify v1.11.1
[...]

I will split the test-go target into test-go-build and test-go-run, so we can measure only the second. Obviously CI users care about both, but equally obviously we care mostly about time to run tests.

Computer (and golang committers) say nope. It is not possible to create a single binary running tests from >1 package. So I'm giving up on this option, and pulling. We will have some numbers: wallclock for how long test-go takes.
And we can improve it later.

[arielshaqed]

THANKS for the great reviews!