Skip to content

StreamResult wasSuccessful ignores unexpectedSuccess while TestResult method fails on any unexpectedSuccess #273

@electrofelix

Description

@electrofelix

StreamResult method wasSuccessful() ignores the value of unexpectedSuccesses in determining whether the overall run was successful or not, however TestResult class does test the value of this attribute.

TestResult wasSuccessful() method result: https://github.com/testing-cabal/testtools/blob/master/testtools/testresult/real.py#L174
StreamResult wasSuccessful method result: https://github.com/testing-cabal/testtools/blob/master/testtools/testresult/real.py#L977

In testing changing StreamResult to align in behaviour with TestResult it appears that tests validating the behaviour against python2.7 fail.

While there are 3 tests that start failing once I changed the behaviour, the one (which is effectively called twice through inheritance) that suggest that it is intentional that StreamResult is defined at
https://github.com/testing-cabal/testtools/blob/master/testtools/tests/test_testresult.py#L223-L229
It results in the following two failures once StreamResult behaviour is changed to treat unexpectedSuccesses as failures:
testtools.tests.test_testresult.TestAdaptedStreamResult.test_addUnexpectedSuccess_was_successful
testtools.tests.test_testresult.TestStreamToExtendedContract.test_addUnexpectedSuccess_was_successful

Upstream python on changed this behaviour to determine the overall test results from unittest being successful sometime around 3.4 - https://bugs.python.org/issue20165

It's unclear if testtools is intentionally maintaining compatibility with unittest's TestResult class prior to this change in it's StreamResult class, or if this is just an oversight and StreamResult should be aligned with TestResult?

Currently this impacts stestr reporting of tests which I've logged as mtreinish/stestr#189 along with noting that it is possible to work around by changing to call the TestResult.wasSuccessful() method passing the StreamResult class instance in as the self argument.

I'm happy to submit a PR if someone can give some guidance on what should be the correct outcome above. Does StreamResult need to remain aligned with the python 2.7 contract, in which any fix needs to do some version detection and different depending on the python version, or is this an oversight in the enhancements that testtools is bringing to python unittests for 2.7?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions