Skip to content

[SPARK-54545][INFRA] Allow test case pattern for run-tests #53254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+15 −1
Open

[SPARK-54545][INFRA] Allow test case pattern for run-tests #53254

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions python/docs/source/development/testing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,19 @@ You can run individual tests by using ``--testnames`` option. For example,

.. code-block:: bash

# Run all tests in a module
python/run-tests --testnames pyspark.sql.tests.test_dataframe
# Run a specific test case
python/run-tests --testnames pyspark.sql.tests.test_dataframe.DataFrameTests.test_range
# Equivalent to the above
python/run-tests --testnames "pyspark.sql.tests.test_dataframe DataFrameTests.test_range"
# You don't need to specify everything, run-tests will do a pattern match
# It uses "-k" for unittest
python/run-tests --testnames "pyspark.sql.tests.test_dataframe test_range"
# This works too
python/run-tests --testnames pyspark.sql.tests.test_dataframe.test_range
# This will match all tests with "range" in their names
python/run-tests --testnames pyspark.sql.tests.test_dataframe.range

Note that you may set ``OBJC_DISABLE_INITIALIZE_FORK_SAFETY`` environment variable to ``YES`` if you are running tests on Mac OS.

Expand Down
5 changes: 4 additions & 1 deletion python/run-tests.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,11 @@ def run_individual_python_test(target_dir, test_name, pyspark_python, keep_test_
"Starting test(%s): %s (temp output: %s)", pyspark_python, test_name, per_test_output.name)
start_time = time.time()
try:
args = test_name.split()
if len(args) == 2:
args.insert(1, "-k")
Copy link
Contributor

@zhengruifeng zhengruifeng Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why insert -k here?
I usually explicitly add -k if needed

Copy link
Contributor Author

@gaogaotiantian gaogaotiantian Nov 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's more like a syntax sugar. Not many people are aware of the existence of -k - we also never mention it in our docs. It's not clear that we are passing the arguments to unittest.main.

The whole purpose of this is to make it easier for people to run an individual test, in an intuitive way.

retcode = subprocess.Popen(
[os.path.join(SPARK_HOME, "bin/pyspark")] + test_name.split(),
[os.path.join(SPARK_HOME, "bin/pyspark")] + args,
stderr=per_test_output, stdout=per_test_output, env=env).wait()
if not keep_test_output:
# There exists a race condition in Python and it causes flakiness in MacOS
Expand Down