Simplecov json report fails to process

[spalmurray]

See thread in codecov/feedback#726 for example report that will have this bug looks like the key is "coverage" not "files" as we expect.

Note that this isn't reproducible without either fixing #3591 or a manual override (comment our VOneProcessor in json processor list).

It seems our simplecov format is outdated. We may need to support multiple versions. Maybe check usage of simplecov though, it might be possible to do a breaking change if nobody uses the current implementation. A quick look at the last 5 years of simplecov changelog didn't reveal anything relevant, so maybe this was just always broken...

worker-1  |  File "/app/apps/worker/services/report/languages/simplecov.py", line 38, in from_json
worker-1  |     for data in json["files"]:
worker-1  |                 ~~~~^^^^^^^^^
worker-1  | KeyError: 'files'

Edit: Simplecov is not necessarily JSON. HTML is the default format, but many formats are supported. Our current docs suggest cobertura and lcov with simplecov, and both of these seem to work. This issue concerns specifically json. See the above thread for details.

[pboling]

@spalmurray Oddly, although it is JSON that was breaking here, what fixed it for me was turning off the HTML output. I kept the JSON on. So perhaps that change resulted in a different order of operations, such that it was able to get to the lcov or rcov sooner, before it choked on the JSON?

Update: Had my wires crossed. Turning off the JSON export fixes it, so JSON parsing truly is broken. See new comment 👇

[pboling]

Regarding JSON here are the relevant libraries:

• ❌ simplecov-json: dead for 5 years, currently very broken, but historically popular, and likely its output is what you built your tool against.
• ✅ simplecov_json_formatter: barely alive (consider sponsoring the authors/maintainers - which is a humble plug, as I've been keeping it alive for the last bit!) This is the gem you should look at when considering output.

[spalmurray]

Huh okay. Let me see if I can figure out what's going on here.

[spalmurray]

In the case of the PR you linked, the simplecov json IS being uploaded, but it's being matched as our V1 format instead of Simplecov. But it still succeeds because it is technically a valid V1 (different json format) report! 😆 The issue that prevented this from working in omniauth-identity is there are lines with the coverage value "ignored", which is not compatible with V1. This may not make sense to you, but should be good context for this issue anyway.

I'm now wondering why removing HTML fixed this repo. Will see if I can find a broken commit from before this PR to look at. Edit: it seems like maybe that repo was never uploading coverage files?

[pboling]

Haha, that may be... I'm about to try the fix on other repos.

[pboling]

Also, it would be cool if it could add the outputs to a stack ranked queue for processing. If one fails, and there is another option on the queue then it tries the next one?

It seems that the JSON format has been turbulent, so perhaps parsing one of the others when present, as a higher precedence, would result in less explosions... I'm conflcited though, because I like explosions.

[spalmurray]

😂 that is definitely a reasonable suggestion. Perhaps it could be controlled in our YAML config continue_on_failure or something CC @rohan-at-sentry

[pboling]

Had my wires crossed. Turning off the JSON export fixes it. So the JSON parsing needs to be fixed.