Broken JUnit-like XML report files

[Paebbels]

Describe the bug

The generated JUnit-like XML file does not conform to any JUnit dialect (Ant JUnit, pytest, GoogleTest, CTest, ...). Despite a missing standard definition for the JUnit XML format, it even violates what many frameworks have agreed on! That said, I also would like to add this situation is disappointing to me, as a project like Bandit under the hood of PyCQA is allowed to produced such outputs without any aspiration for correctness...

To get forward I'll give a review of all of my findings and a proposal how the XML output can be fixed or should look like. I'm not the first one complaining about the unusable output format, because "no" other tool can read the produced output file. See also:

• missing expected elements (e.g. errors, failures, testcase list, ... ) if no failures in xml output file #634
• XML (JUnit) output is not recognized by Bamboo JUnit parser #712

Here is my review of all findings:

1. Missing XML root element <testsuites>.
   This is for debate, as some other dialects also omitt this element.
   E.g. pytest used testsuites → testsuite → testcase.

2. Attributes errors="0", failures="0", skipped="0" are missing. In case failed tests get listed, failures must be non-zero. See also 8.

3. Attribute time is missing on tag testsuite to denote the accumulated runtime for a testsuite.

4. Attribute timestamp is missing on tag testsuite to denote when the checks were run.

5. Attribute classname is misused for the filename. Attribute file must be used for this purpose.
   Use the Python package or module name for this field.

6. Attribute name is misused for a category of tests. A category could be encoded either as a testcase property or by using multiple testsuites - one testsuite per category.
   Use the name of the test for this field. E.g. B404-import-subprocess`

7. The line number information is hidden in the text.
   Attribute line should be set for this purpose.

8. Using an error tag for failed bandit tests is wrong.
   Tag failure must be used in this case!
   A not passing assertion is a failure, not an error according to JUnit.
   Errors are reserved for test abortions like syntax errors, import errors, raised exceptions when inspecting the code, etc.

9. Attribute more_info in tag error isn't described in any JUnit dialect and will cause an XML schema validation error.
   Use properties to express additional information.

10. The message tries to list multiple information line-by-line, but some information is presented on the same text line without a delimiter.
    Moreover, the issue description has no "header" and Location misses a colon.
    This is not fulfilling the criteria for machine readable outputs!
    Keep in mind, the Ant JUnit XML format is a data exchange format from tool to tool, not a visualization for humans.

    Improvements for a better machine readable text message:

    Test ID: B404
    Severity: LOW
    Confidence: HIGH
    CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
    Description: Consider possible security implications associated with the subprocess module.
    Location: myPackage\__init__.py:43
    

11. Information should be provided as properties, thus machines can easily read (extract) and filter testcases:

    <properties>
      <property name="Test-ID" value="B404" />
      <property name="Severity" value="Low" />
      <property name="Confidence" value="High" />
      <property name="CWE-ID" value="CWE-78" />
      <property name="CWE-URL" value="https://cwe.mitre.org/data/definitions/78.html" />
      <property name="CEW-Description">
        Consider possible security implications associated with the subprocess module.
      </property>
      <property name="Bandit-URL" value="https://bandit.readthedocs.io/en/1.8.6/blacklists/blacklist_imports.html#b404-import-subprocess" />
    </properties>

12. When multiple issues are discovered for the same file, the tuple of attributes classname and name are identical.
    The JUnit format requires them to be unique.

Summarizing all findings and proposed fixes, a JUnit conforming output could look like this:

<?xml version='1.0' encoding='utf-8'?>
<testsuite name="bandit" tests="1" failures="1" errors="0" skipped="0" time="1.2" timestamp="2025-09-25T09:50:52+02:00">
	<testcase classname="myPackage" name="B404-import-subprocess" file="myPackage\__init__.py" line="43" time="0.2">
		<properties>
			<property name="Test-ID" value="B404" />
			<property name="Test-Name" value="import-subprocess" />
			<property name="Severity" value="Low" />
			<property name="Confidence" value="High" />
			<property name="CWE-ID" value="CWE-78" />
			<property name="CWE-URL" value="https://cwe.mitre.org/data/definitions/78.html" />
			<property name="CEW-Description">Consider possible security implications associated with the subprocess module.</property>
			<property name="Bandit-URL" value="https://bandit.readthedocs.io/en/1.8.6/blacklists/blacklist_imports.html#b404-import-subprocess" />
		</properties>
		<failure message="Consider possible security implications associated with the subprocess module.">Test ID: B404
			Severity: LOW
			Confidence: HIGH
			CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
			Description: Consider possible security implications associated with the subprocess module.
			Location: myPackage\__init__.py:43
		</failure>
	</testcase>
</testsuite>

Following closer to the XML format used by pytest, it would look like this:

<?xml version='1.0' encoding='utf-8'?>
<testsuites name="bandit">
	<testsuite name="bandit" tests="1" failures="1" errors="0" skipped="0" time="1.2" timestamp="2025-09-25T09:50:52+02:00" hostname="">
		<properties>
			<property name="Bandit-Version" value="1.8.6" />
		</properties>
		<testcase classname="myPackage" name="B404-import-subprocess" file="myPackage\__init__.py" line="43" time="0.2">
			<properties>
				<property name="Test-ID" value="B404" />
				<property name="Test-Name" value="import-subprocess" />
				<property name="Severity" value="Low" />
				<property name="Confidence" value="High" />
				<property name="CWE-ID" value="CWE-78" />
				<property name="CWE-URL" value="https://cwe.mitre.org/data/definitions/78.html" />
				<property name="CEW-Description">Consider possible security implications associated with the subprocess module.</property>
				<property name="Bandit-URL" value="https://bandit.readthedocs.io/en/1.8.6/blacklists/blacklist_imports.html#b404-import-subprocess" />
			</properties>
			<failure message="Consider possible security implications associated with the subprocess module.">Test ID: B404
				Severity: LOW
				Confidence: HIGH
				CWE: CWE-78 (https://cwe.mitre.org/data/definitions/78.html)
				Description: Consider possible security implications associated with the subprocess module.
				Location: myPackage\__init__.py:43
			</failure>
		</testcase>
	</testsuite>
</testsuites>

Besides the format itself, there is another big issue.
The JUnit XML report file meaning is inverted. An empty testsuite represents a passed Bandit check, while a filled testsuite represents failed checks.
At first, almost any tool reading a JUnit report will complain if it's empty and emit either an error or at least a warning!
At second, the absence of data (empty report) can not be used to express a passed message, because a bug could also lead to an empty report. Again, absence of data is not an acceptable principle for a passed test information.

How to solve this problem?
Bandit has a fixed list of tests that are performed on every Python package and module:

• https://bandit.readthedocs.io/en/latest/plugins/index.html#complete-test-plugin-listing
• https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html
• https://bandit.readthedocs.io/en/latest/blacklists/blacklist_imports.html#blacklist-imports

Bandit could create one testcase per Bandit test and module.

---

I propose the following steps forward, if wished by the Bandit developers.

1. Fix the XML output format so other tools can read and verify the generated XML files.
   Provide information as machine readable property fields.
2. Tackle the inversion problem.

Reproduction steps

Run Bandit on any Python package containing e.g. import listed on the blacklist and generate an XML report by Bandit.

Expected behavior

• A JUnit report format following the common Ant-JUnit or Pytest-JUnit format.
• A JUnit XML file that can be validated against an XSD.
  See https://github.com/edaa-org/pyEDAA.Reports/tree/main/pyEDAA/Reports/Resources
• Machine readable information.
• No inversion of a JUnit report meaning.

Bandit version

1.8.3 (Default)

Python version

3.13 (Default)

Additional context

This was a review summarized as a list of finding and proposed solutions.

I would like to know if Bandit developers are interested in fixing this problem. If yo I might be willing to propose a PR. But before I would like to understand if my time will be well spent.