Run the functests in reverse on CI

For speed h doesn't clear the DB between functests, see:
#6845

As a defence against tests becoming dependent on data left in the DB by
the tests that run before them, we ran the functests in reverse order on
CI (but in normal order in dev). This was achieved by adding the
`pytest-reverse` dependency and adding `--reverse` to the `pytest`
command in CI. See: #7013

The `--reverse` was later accidentally removed by
#8264. The cookiecutter has since
been applied to h and the cookiecutter doesn't do the `--reverse`
either.

The cookiecutter's `ci.yml` doesn't provide a way for an individual
project to add a command line argument to the `pytest` command for the
functests.

One solution would be to add such an option to the cookiecutter, but
it's awkward and I don't like adding too many options to the
cookiecutter.

Another solution would be to add the `pytest-reverse` dependency and
`--reverse` argument to the cookiecutter and have it be applied to all
projects, and possibly also for the unittests not just the functests.
This could be a good option, but
[pytest-reverse](https://github.com/adamchainz/pytest-reverse) is a
third-party pytest plugin not an official one, and not very popular, so
perhaps we don't want to depend on it in every repo. Also, all our other
projects reset the DB before each unittest or functest so they have no
need for `pytest-reverse`. If we ever fix h to reset the DB before each
functest we'll remove the `pytest-reverse` dependency from h as well.

So this commit instead finds a clever tox way to add the `--reverse`
command line option in CI but not in dev:

1. We add the `--reverse` to the `PYTEST_ADDOPTS` envvar, which is an
   environment variable of command line options that pytest supports as
   an alternative to actually including the options in the command. The
   cookiecutter already supports per-project customisation of the
   envvars in the `tox.ini` file so we can do this just for h without
   having to add any new options to the cookiecutter.
2. To get tox to include `--reverse` in `PYTEST_ADDOPTS` when running on
   CI but not when running in dev we uses tox's [interactive shell
   substitution](https://tox.wiki/en/3.9.0/config.html#interactive-shell-substitution)
   which is a feature that lets you substitute one value when tox is
   running in an interactive TTY (i.e. in dev) and another value when
   it's running non-interactively (i.e. on CI) like so:
   `PYTEST_ADDOPTS = {env:PYTEST_ADDOPTS:{tty::--reverse}}` (in this
   case the non-TTY value is an empty string).

A bit clever perhaps but it gets the job done and it's a one-liner. If
we ever fix h's functests to reset the DB before each test then we can
remove both the `pytest-reverse` dependency and this hack.

Author: seanh

.cookiecutter/includes/tox/setenv

@@ -13,4 +13,5 @@ dev: REPLICA_DATABASE_URL = {env:DATABASE_URL:postgresql://postgres@localhost/po
 dev: MAILCHIMP_USER_ACTIONS_SUBACCOUNT = {env:MAILCHIMP_USER_ACTIONS_SUBACCOUNT:devdata}
 tests: ELASTICSEARCH_INDEX = {env:ELASTICSEARCH_INDEX:hypothesis-tests}
 functests: ELASTICSEARCH_INDEX = {env:ELASTICSEARCH_INDEX:hypothesis-functests}
+functests: PYTEST_ADDOPTS = {env:PYTEST_ADDOPTS:{tty::--reverse}}
 {tests,functests}: AUTHORITY = {env:AUTHORITY:example.com}

pyproject.toml

@@ -1,5 +1,4 @@
 [tool.pytest.ini_options]
-addopts = "-q"
 filterwarnings = [
     "error", # Fail the tests if there are any warnings.
     "ignore:^find_module\\(\\) is deprecated and slated for removal in Python 3.12; use find_spec\\(\\) instead$:DeprecationWarning:importlib",

tox.ini

@@ -37,6 +37,7 @@ setenv =
     dev: MAILCHIMP_USER_ACTIONS_SUBACCOUNT = {env:MAILCHIMP_USER_ACTIONS_SUBACCOUNT:devdata}
     tests: ELASTICSEARCH_INDEX = {env:ELASTICSEARCH_INDEX:hypothesis-tests}
     functests: ELASTICSEARCH_INDEX = {env:ELASTICSEARCH_INDEX:hypothesis-functests}
+    functests: PYTEST_ADDOPTS = {env:PYTEST_ADDOPTS:{tty::--reverse}}
     {tests,functests}: AUTHORITY = {env:AUTHORITY:example.com}
 passenv =
     HOME
@@ -85,7 +86,7 @@ commands =
     lint: {posargs:ruff check h tests bin}
     {tests,functests}: python3 -m h.scripts.init_db --delete --create
     tests: python -m pytest --cov --cov-report= --cov-fail-under=0 {posargs:--numprocesses logical --dist loadgroup tests/unit/}
-    functests: python -m pytest --failed-first --new-first --no-header --quiet {posargs:tests/functional/}
+    functests: python -m pytest {posargs:tests/functional/}
     coverage: coverage combine
     coverage: coverage report
     typecheck: mypy h