Skip to content

Align method filtering behavior between JUnit and TestNG#3227

Closed
pdolif wants to merge 1 commit intoapache:masterfrom
pdolif:align-method-filtering-junit-testng
Closed

Align method filtering behavior between JUnit and TestNG#3227
pdolif wants to merge 1 commit intoapache:masterfrom
pdolif:align-method-filtering-junit-testng

Conversation

@pdolif
Copy link

@pdolif pdolif commented Nov 27, 2025

Closes #3225
This PR aligns the method filtering behavior between JUnit and TestNG by adjusting the TestNG MethodSelector.

Problem

Given the following test classes:

public class SubClass extends SuperClass {
    @Test
    public void testInSubClass() {}
}
public abstract class SuperClass {
    @Test
    public void testInSuperClass() {}
}
public class OtherClass {
    @Test
    public void testIncluded() {}

    @Test
    public void testNotIncluded() {}
}

When using an includesFile containing SubClass only, testInSubClass and testInSuperClass are executed (as expected).

If the includesFile does not only contain class filters but also method filters, the behavior is different (with TestNG only), and testInSuperClass is not executed anymore.
Example includesFile:

SubClass
OtherClass#testIncluded

With this, only testInSubClass and OtherClass#testIncluded are executed.

Root Cause

TestNG uses the MethodSelector for filtering the tests. For testInSuperClass() the MethodSelector is invoked with an ITestNGMethod instance like this:

  • getRealClass() returns SuperClass
  • getTestClass().getRealClass() returns SubClass

The current MethodSelector implementation uses getRealClass() and then checks whether to run SuperClass#testInSuperClass. If we have SubClass in the includesFile, the test will not be executed, even though SubClass inherits it. This is different when using JUnit instead of TestNG.

Fix

This PR changes the MethodSelector implementation to use getTestClass().getRealClass() to check whether to run a test. The behavior changes as follows:

  • In the example above, testInSuperClass gets executed too.
  • With SubClass#testInSuperClass in the includesFile, prior to this PR no test was executed. Now SubClass#testInSuperClass gets executed.
  • With SubClass in the includesFile and SuperClass#testInSuperClass in the excludesFile, prior to this PR only SubClass#testInSubClass got executed. Now the inherited test SubClass#testInSuperClass gets executed too.

The PR includes tests that are executed with TestNG and JUnit to ensure the behavior aligns.


Following this checklist to help us incorporate your
contribution quickly and easily:

  • Each commit in the pull request should have a meaningful subject line and body.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Run mvn clean install to make sure basic checks pass. A more thorough check will
    be performed on your pull request automatically.
  • You have run the integration tests successfully (mvn -Prun-its clean install).

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.

To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

@olamy
Copy link
Member

olamy commented Nov 27, 2025

Could you please check with this PR #3179 as the big refactoring might land very soon now

@pdolif
Copy link
Author

pdolif commented Nov 29, 2025

@olamy The integration test added in this PR succeeds with PR #3179.
I originally noticed the behavior described in #3225 when having a look at apache/pulsar#24921. Now I tried to run the Pulsar tests with the snapshot version of #3179 but it skips a lot of tests (~900 of ~1,700 in "broker" group).

@Tibor17
Copy link
Contributor

Tibor17 commented Jan 28, 2026

@juherr is developer at TestNG.
I need to have his inside into this work.
@juherr can you please have a look? The logic seems to be strange. There is named "Fake..." class, etc. Is this trusty PR?

@olamy
Copy link
Member

olamy commented Jan 29, 2026

this is typically the kind of duplicated code which will not be needed anymore with #3179

@juherr
Copy link
Contributor

juherr commented Jan 29, 2026

@Tibor17 At the moment, it works without breaking anything. To me, FakeClass doesn’t sound any stranger than FakeTestMethod.

@pdolif pdolif closed this Mar 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestNG: Inherited tests are not executed when includesFile contains method filters

4 participants