feat(cli): add `filter_fixtures` cli command #2045

danceratopz · 2025-08-15T11:33:52Z

🗒️ Description

Implements a new CLI command to filter fixture files by fork, enabling creation of stable release sets from development fixtures.

Example Usage

Filter out Osaka, ToOsaka transition forks (and newer forks):

uv run filter_fixtures --input=fixtures_develop --output=fixtures_stable --until=Prague --clean

Filter specific fork only

uv run filter_fixtures --input=fixtures --output=cancun_only --fork=Cancun

CI Release Workflow

This cli tool will be a critical element of the fixture release flow. The unit test pytest src/cli/tests/test_filter_fixtures.py::test_ci_release_validation aims to allow sanity checking of the filtered fixture output directory fixtures_stable and provides verbose output for easy manual verification.

Integration in CI is to be done in a follow-up PR.

The idea is:

Generate fixtures including development fork

fill --output=fixtures_develop --until=Osaka

Create stable release set (exclude development fork)

uv run filter_fixtures --input=fixtures_develop --output=fixtures_stable --until=Prague --quiet --clean

Validate filtering results

uv run pytest src/cli/tests/test_filter_fixtures.py::test_ci_release_validation \
  --before-fixtures-dir=fixtures_develop \
  --after-fixtures-dir=fixtures_stable -s

Example output (with unit tests instead of release data):

uv run pytest src/cli/tests/test_filter_fixtures.py::test_ci_release_validation -s 
src/cli/tests/test_filter_fixtures.py 🔧 Auto-setup: Running filter_fixtures for unit testing...
   Before: test_fixtures/ (all forks)
   After:  tmp8axmphap/ (filtered --until Prague)


============================================================
CI Release Validation Report
============================================================

📁 Directories:
  Before (develop): test_fixtures/
  After (stable):   tmp8axmphap/

🍴 Fork Analysis (Chronological Order):
------------------------------------------------------------
Fork                           Develop  Stable   Status    
------------------------------------------------------------
Byzantium                      2        2        ✅ Kept    
ConstantinopleFix              2        2        ✅ Kept    
Istanbul                       2        2        ✅ Kept    
Berlin                         2        2        ✅ Kept    
London                         2        2        ✅ Kept    
Paris                          4        4        ✅ Kept    
ParisToShanghaiAtTime15k       3        3        ✅ Kept    
Shanghai                       4        4        ✅ Kept    
ShanghaiToCancunAtTime15k      3        3        ✅ Kept    
Cancun                         4        4        ✅ Kept    
CancunToPragueAtTime15k        3        3        ✅ Kept    
Prague                         4        4        ✅ Kept    
PragueToOsakaAtTime15k         3        0        ❌ Filtered
Osaka                          4        0        ❌ Filtered
------------------------------------------------------------
TOTAL                          42       35         83.3% retained

✂️ Filtered Out (in chronological order):
  - PragueToOsakaAtTime15k (3 tests)
  - Osaka (4 tests)

✅ Validation Results:
  - Stable is subset of develop: PASS
  - Test count consistency: PASS
  - Fork count consistency: PASS
  - Meaningful filtering: PASS

🔗 Related Issues or PRs

Partially resolves:

chore(fill,ci): only generate each fixture once per release #2039

✅ Checklist

All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
```
uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
```
All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
All: Considered adding an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
All: Set appropriate labels for the changes (only maintainers can apply labels).

This funtion was added to handle transition forks.

danceratopz · 2025-08-15T11:51:22Z

src/cli/tests/test_filter_fixtures.py

+    with before_index_path.open() as f:
+        before_index = json.load(f)
+
+    with after_index_path.open() as f:
+        after_index = json.load(f)


This test function is critical to verify releases in CI: It should be ran (see docstring/PR description) to compare the original (fixtures_develop) and filtered output (fixtures_stable) to automatically sanity check them and provide console output for an easy manual visual check.

Note, this is only validating via index files. filter_fixtures will generate a new index.json for the new fixtures output dir via genindex. But, in a way, the verification here is a "light" verification. Perhaps we should do a full verification by loading all fixture files ourselves here, but perhaps not, I mean this is what genindex does anyway... Will leave as-is for now. Happy to hear your feedback.

marioevz

LGTM, we should first merge #2051, apply the comments, and then merge this one, because otherwise the logic to select the forks is going to differ from the logic that we use in fill.

marioevz · 2025-08-19T17:37:50Z

src/cli/filter_fixtures.py

+@click.option("--fork", "single_fork", help="Only include fixtures for the specified fork.")
+@click.option(
+    "--from", "forks_from", help="Include fixtures from and including the specified fork."
+)
+@click.option(
+    "--until", "forks_until", help="Include fixtures until and including the specified fork."
+)


We can use ForkSetAdapter from #2051 to make this a bit easier.

Then:

Suggested change

@click.option("--fork", "single_fork", help="Only include fixtures for the specified fork.")

@click.option(

"--from", "forks_from", help="Include fixtures from and including the specified fork."

)

@click.option(

"--until", "forks_until", help="Include fixtures until and including the specified fork."

)

@click.option(

"--fork",

"single_fork",

type=ForkSetAdapter.validate_python,

default=set(),

help="Only include fixtures for the specified fork.",

)

@click.option(

"--from",

"from_fork",

type=ForkSetAdapter.validate_python,

default=set(),

help="Include fixtures from and including the specified fork.",

)

@click.option(

"--until",

"until_fork",

type=ForkSetAdapter.validate_python,

default=set(),

help="Include fixtures until and including the specified fork.",

)

marioevz · 2025-08-20T01:26:13Z

src/cli/filter_fixtures.py

+        single_fork_obj, from_fork, until_fork = get_valid_fork_range(
+            single_fork, forks_from, forks_until
+        )


Can be removed with the previous comment.

marioevz · 2025-08-20T01:27:26Z

src/cli/filter_fixtures.py

+def should_include_fixture_fork(
+    fixture_fork: Type[BaseFork],
+    single_fork: Type[BaseFork] | None,
+    from_fork: Type[BaseFork] | None,
+    until_fork: Type[BaseFork] | None,
+) -> bool:
+    """
+    Determine if a fixture's fork should be included based on the filtering criteria.
+
+    This uses fork comparison operators to handle both regular and transition forks.
+    """
+    if single_fork:
+        return fixture_fork == single_fork
+
+    # Default range is all deployed forks if no range specified
+    if not from_fork and not until_fork:
+        deployed_forks = get_deployed_forks()
+        return deployed_forks[0] <= fixture_fork <= deployed_forks[-1]
+
+    # Handle range filtering
+    from_ok = True
+    until_ok = True
+
+    if from_fork:
+        from_ok = fixture_fork >= from_fork
+
+    if until_fork:
+        until_ok = fixture_fork <= until_fork
+
+    return from_ok and until_ok


We should use the get_selected_fork_set function introduced in #2051 to match the exact logic that fill uses.

The result of that function is a set, so you can do fork in get_selected_fork_set(...).

marioevz · 2025-08-21T17:19:24Z

src/cli/tests/test_fixtures/.meta/index.json

This whole folder can be generated on the fly. IMO we should not be adding this much static files for an unit test for a subcommand.

marioevz · 2025-08-21T17:29:39Z

src/cli/tests/test_filter_fixtures.py

+    all_forks_chronological = get_all_forks_chronologically()
+    fork_names_ordered = [fork.name() for fork in all_forks_chronological]
+
+    # Filter to only forks present in either directory
+    relevant_forks = [
+        fork_name
+        for fork_name in fork_names_ordered
+        if fork_name in before_forks or fork_name in after_forks
+    ]


All of this seems unnecessary (including the new get_all_forks_chronologically).
Let's remove most of the fork logic in here now and just reuse the ForkSetAdapter.validate_python(config_str) and get_selected_fork_set in here:

execution-spec-tests/src/pytest_plugins/forks/forks.py

Lines 454 to 479 in 961cdc8

def get_fork_option(config, option_name: str, parameter_name: str) -> Set[Fork]:

"""Post-process get option to allow for external fork conditions."""

config_str = config.getoption(option_name)

try:

return ForkSetAdapter.validate_python(config_str)

except InvalidForkError:

print(

f"Error: Unsupported fork provided to {parameter_name}:",

config_str,

"\n",

file=sys.stderr,

)

print(available_forks_help, file=sys.stderr)

pytest.exit("Invalid command-line options.", returncode=pytest.ExitCode.USAGE_ERROR)

single_fork = get_fork_option(config, "single_fork", "--fork")

forks_from = get_fork_option(config, "forks_from", "--from")

forks_until = get_fork_option(config, "forks_until", "--until")

show_fork_help = config.getoption("show_fork_help")

dev_forks_help = textwrap.dedent(

"To run tests for a fork under active development, it must be "

"specified explicitly via --until=FORK.\n"

"Tests are only ran for deployed mainnet forks by default, i.e., "

f"until {get_deployed_forks()[-1].name()}.\n"

)

danceratopz added 4 commits August 15, 2025 13:09

feat(forks): add helper function to print all forks chronologically

91bbb4b

This funtion was added to handle transition forks.

feat: add filter_fixtures (by fork) cli command

5f43bf8

tests: add unit tests for filter_fixtures

e3ba7b4

docs: update changelog

63754c2

danceratopz added type:feat type: Feature scope:cli Scope: Changes to `ethereum_clis` labels Aug 15, 2025

danceratopz commented Aug 15, 2025

View reviewed changes

felix314159 mentioned this pull request Aug 18, 2025

chore(fill,ci): only generate each fixture once per release #2039

Open

marioevz mentioned this pull request Aug 20, 2025

refactor(forks): refactor extract fork until from logic #2051

Merged

4 tasks

marioevz reviewed Aug 20, 2025

View reviewed changes

marioevz requested changes Aug 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(cli): add `filter_fixtures` cli command #2045

feat(cli): add `filter_fixtures` cli command #2045

Uh oh!

danceratopz commented Aug 15, 2025

Uh oh!

danceratopz Aug 15, 2025

Uh oh!

marioevz left a comment

Uh oh!

marioevz Aug 19, 2025

Uh oh!

marioevz Aug 20, 2025

Uh oh!

marioevz Aug 20, 2025

Uh oh!

marioevz Aug 21, 2025

Uh oh!

marioevz Aug 21, 2025

Uh oh!

Uh oh!

	def get_fork_option(config, option_name: str, parameter_name: str) -> Set[Fork]:
	"""Post-process get option to allow for external fork conditions."""
	config_str = config.getoption(option_name)
	try:
	return ForkSetAdapter.validate_python(config_str)
	except InvalidForkError:
	print(
	f"Error: Unsupported fork provided to {parameter_name}:",
	config_str,
	"\n",
	file=sys.stderr,
	)
	print(available_forks_help, file=sys.stderr)
	pytest.exit("Invalid command-line options.", returncode=pytest.ExitCode.USAGE_ERROR)

	single_fork = get_fork_option(config, "single_fork", "--fork")
	forks_from = get_fork_option(config, "forks_from", "--from")
	forks_until = get_fork_option(config, "forks_until", "--until")
	show_fork_help = config.getoption("show_fork_help")

	dev_forks_help = textwrap.dedent(
	"To run tests for a fork under active development, it must be "
	"specified explicitly via --until=FORK.\n"
	"Tests are only ran for deployed mainnet forks by default, i.e., "
	f"until {get_deployed_forks()[-1].name()}.\n"
	)

feat(cli): add filter_fixtures cli command #2045

Are you sure you want to change the base?

feat(cli): add filter_fixtures cli command #2045

Uh oh!

Conversation

danceratopz commented Aug 15, 2025

🗒️ Description

Example Usage

CI Release Workflow

🔗 Related Issues or PRs

✅ Checklist

Uh oh!

danceratopz Aug 15, 2025

Choose a reason for hiding this comment

Uh oh!

marioevz left a comment

Choose a reason for hiding this comment

Uh oh!

marioevz Aug 19, 2025

Choose a reason for hiding this comment

Uh oh!

marioevz Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

marioevz Aug 20, 2025

Choose a reason for hiding this comment

Uh oh!

marioevz Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

marioevz Aug 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feat(cli): add `filter_fixtures` cli command #2045

feat(cli): add `filter_fixtures` cli command #2045