test isolation: $RUFF_OUTPUT_FORMAT and ~/.config/ruff/pyproject.toml

[tsvikas]

when running the test suite using cargo test, i noticed that my personal configuration from
$RUFF_OUTPUT_FORMAT and ~/.config/ruff/pyproject.toml leaked and effected the tests (i.e. some snapshots were different)

is it intended? i can assume that the 'official' way to run tests is in docker containers, but than

1. it's slower
2. we might want to say it in https://docs.astral.sh/ruff/contributing .

i'm not a ruff programmer so i don't know if there is a way to isolate the test suite from those effects, but i assume there might be.

just something to improve the experience of new contributors.

also , i got :

        FAIL [   0.081s] ruff_linter rules::flake8_executable::tests::rules::path_new_exe001_1_py_expects
        FAIL [   0.091s] ruff_linter rules::flake8_executable::tests::rules::path_new_exe002_1_py_expects

on the main branch, and i'm not sure why. is it something in the cloning? or is it failing on main for everyone?

[ntBre]

Thanks for opening this! I've definitely run into test failures when I had a ruff.toml file in my /tmp directory, so I can relate. I guess most of us just don't have global ruff configs because I don't think we normally run tests in containers (I know I don't at least).

Something like:

env -u RUFF_OUTPUT_FORMAT cargo test ...

seems to work well enough for the output format, but I haven't been able to find a nice env command for avoiding the global config discovery. Unsetting $HOME doesn't seem to work.

I'm only seeing 5 failures from a (small) global ruff config file:

        FAIL [   0.163s] ruff::cli lint::requires_python_no_tool_preview_enabled
        FAIL [   0.158s] ruff::cli lint::requires_python_pyproject_toml_above
        FAIL [   0.181s] ruff::cli lint::requires_python_no_tool_target_version_override
        FAIL [   0.204s] ruff::cli lint::requires_python_no_tool
        FAIL [   0.173s] ruff::cli lint::requires_python_pyproject_toml_above_with_tool

Is that what you're seeing too? We might be able just to fix these tests, especially since they all seem related.

I'm not seeing the EXE test failures locally or in CI, so that might be related to your environment too.

[MichaReiser]

The fix for the CLI tests is relatively easy, we simply need to reset the environment variables

ruff/crates/ruff/tests/cli/main.rs

Line 193 in 8ccda14

command.env_clear();

We could add the clear_env call as a mitigation but the long term solution is to migrate all cli tests to use CliTest

[tsvikas]

Apart from the EXE test failures:

• those tests fail if RUFF_OUTPUT_FORMAT=concise:

        FAIL [   0.095s] ruff::integration_test check_extend_unsafe_fixes
        FAIL [   0.102s] ruff::integration_test check_extend_safe_fixes
        FAIL [   0.101s] ruff::integration_test check_extend_unsafe_fixes_conflict_with_extend_safe_fixes
        FAIL [   0.128s] ruff::integration_test check_default_files
        FAIL [   0.099s] ruff::integration_test check_hints_hidden_unsafe_fixes
        FAIL [   0.123s] ruff::integration_test check_docstring_conventions_overrides
        FAIL [   0.119s] ruff::integration_test check_extend_unsafe_fixes_conflict_with_extend_safe_fixes_by_specificity
        FAIL [   0.096s] ruff::integration_test check_no_hint_for_hidden_unsafe_fixes_when_disabled
        FAIL [   0.094s] ruff::integration_test check_no_hint_for_hidden_unsafe_fixes_with_no_safe_fixes_when_disabled
        FAIL [   0.092s] ruff::integration_test check_shows_unsafe_fixes_with_opt_in
        FAIL [   0.116s] ruff::integration_test check_hints_hidden_unsafe_fixes_with_no_safe_fixes
        FAIL [   0.108s] ruff::integration_test check_warn_stdin_filename_with_files
        FAIL [   0.138s] ruff::integration_test check_input_from_argfile
        FAIL [   0.106s] ruff::integration_test deprecated_direct
        FAIL [   0.109s] ruff::integration_test deprecated_multiple_direct
        FAIL [   0.093s] ruff::integration_test fix_does_not_apply_display_only_fixes
        FAIL [   0.102s] ruff::integration_test fix_applies_safe_fixes_by_default
        FAIL [   0.105s] ruff::integration_test fix_does_not_apply_display_only_fixes_with_unsafe_fixes_enabled
        FAIL [   0.105s] ruff::integration_test fix_only_unsafe_fixes_available
        FAIL [   0.094s] ruff::integration_test fix_preview
        FAIL [   0.101s] ruff::integration_test full_output_preview
        FAIL [   0.101s] ruff::integration_test full_output_preview_config
        FAIL [   0.104s] ruff::integration_test parse_error_not_included
        FAIL [   0.098s] ruff::integration_test pyproject_toml_stdin_schema_error
        FAIL [   0.098s] ruff::integration_test pyproject_toml_stdin_schema_error_fix
        FAIL [   0.092s] ruff::integration_test redirect_direct
        FAIL [   0.101s] ruff::integration_test pyproject_toml_stdin_syntax_error
        FAIL [   0.103s] ruff::integration_test redirect_prefix
        FAIL [   0.101s] ruff::integration_test stdin_error
        FAIL [   0.107s] ruff::integration_test stdin_filename
        FAIL [   0.113s] ruff::integration_test stdin_fix_jupyter
        FAIL [   0.101s] ruff::integration_test stdin_fix_when_not_fixable_should_still_print_contents
        FAIL [   0.098s] ruff::integration_test stdin_multiple_parse_error
        FAIL [   0.108s] ruff::integration_test stdin_format_jupyter
        FAIL [   0.106s] ruff::integration_test stdin_override_parser_ipynb
        FAIL [   0.092s] ruff::integration_test stdin_parse_error
        FAIL [   0.097s] ruff::integration_test stdin_override_parser_py
        FAIL [   0.104s] ruff::integration_test stdin_source_type_py
        FAIL [   0.112s] ruff::integration_test unfixable_preview
        FAIL [   0.152s] ruff::show_settings display_default_settings
        FAIL [   0.030s] ruff_dev::bin/ruff_dev generate_cli_help::tests::test_generate_json_schema

• and this ~.config/ruff/pyproject.toml`

[tool.ruff]
target-version = "py313"

[tool.ruff.lint]
# from https://docs.astral.sh/ruff/linter/#rule-selection
select = [
  # pycodestyle
  "E",
  # Pyflakes
  "F",
  # pyupgrade
  "UP",
  # flake8-bugbear
  "B",
  # flake8-simplify
  "SIM",
  # isort
  "I",
]
ignore = [
  # incompatible with other rules:
  "COM812",   # flake8-commas/missing-trailing-comma
  # disagree:
  "PD010",    # pandas-vet/pandas-use-of-dot-pivot-or-unstack
  "PD013",    # pandas-vet/pandas-use-of-dot-stack
  # annoying:
  "EM10",     # flake8-errmsg/*-in-exception
  "RET504",   # flake8-return/unnecessary-assign
]

[tool.ruff.format]
docstring-code-format = true

affect these tests:

        FAIL [   0.190s] ruff::cli lint::requires_python_no_tool_target_version_override
        FAIL [   0.173s] ruff::cli lint::requires_python_pyproject_toml_above_with_tool
        FAIL [   0.173s] ruff::cli lint::requires_python_pyproject_toml_above
        FAIL [   0.215s] ruff::cli lint::requires_python_no_tool
        FAIL [   0.203s] ruff::cli lint::requires_python_no_tool_preview_enabled
        FAIL [   0.100s] ruff::cli lint::walrus_before_py38