MzIdentML Validation Feature

[sureshhewabi]

• 1. Validation of Crosslinking MzIdentML (mzID) files.

  • a. Validate files of a given folder(Input will be file path)
  • b. Multiple MzIdentML files
  • c. Individual MzIdentML file
  • d. Schema Validations
  • e. Semantic Validations(If any)
  • f. References are valid
  • g. Other validations(eg: No of Protein, peptide, spectra)

• 2. Generate validation report
  We can generate a simple report consisting above information

This is a simple start, and let's use this issue to discuss on validation

[colin-combe]

one issue here is that to fully validate the mzIdentML file you also need the peaklists, e.g. #81

[colin-combe]

what are peoples views on how to deal with that? Two alternatives would be:

• allow partial validation without peaklists. PRO: people don't have to deal with moving the peaklist files around. CON: later it could all turn out to be broken
• require peaklists are provided in order to validate. PRO: you know its right; CON: peaklist files

[colin-combe]

what if we concentrate on (a) 'Validate files of a given folder(Input will be file path)',
and this folder must also contain the peaklist files?

This is easiest to do because its most like how the converter already works.

Also, if it just stops after the first error, then that's easier.

Thoughts on this?

[colin-combe]

@sureshhewabi - #82 - you can take a look at what I've done there

that PR gives a command line validation option.

• arg can be directory or specific file - https://github.com/Rappsilber-Laboratory/xi-mzidentml-converter/blob/pride/parser/process_dataset.py#L40-L47
• it requires peaklists alongside mzId files
• the code it uses for schema validation is here - https://github.com/Rappsilber-Laboratory/xi-mzidentml-converter/blob/pride/parser/schema_validate.py
• if schema validation passes it creates a sqlite db in temp folder and loads data into it - https://github.com/Rappsilber-Laboratory/xi-mzidentml-converter/blob/pride/parser/process_dataset.py#L281-L326
• any errors cause it to print out error and abort

So, as a first attempt, i think covers 1. (a), (c), (d), (e) to very a limited extent, and (f) above.
1.(b) we could live without in short term. 1 (g), as i read it, isn't really validation but summary stats, these could be got by querying the sqlite DB.

For 2. above, info is printed to standard output, think it currently includes the logging info we usually see from the converter.

Its not extensively tested. It passes the file Diogo provided. It fails the schema invalid Kojak file.

[sureshhewabi]

what if we concentrate on (a) 'Validate files of a given folder(Input will be file path)', and this folder must also contain the peaklist files?

This is easiest to do because its most like how the converter already works.

Also, if it just stops after the first error, then that's easier.

Thoughts on this?

Yes I agree with that

[colin-combe]

I agree with that

good, that's the way it works in that PR

[colin-combe]

Its not currently rejecting files that don't have the sequences in Seq elements.
(That additional requirement of ours.)
It means they break later. (Also of no use to PDB-IHN without sequences?)
I'll need to change so it rejects these.

[colin-combe]

Also, I think I've found another requirement specific to our system - that all Modifications have masses given.

[colin-combe]

Also, I think I've found another requirement specific to our system - that all Modifications have masses given.

hmm, i think we shouldn't add that as a requirement, rather the spectrum viewer is broken in some cases at the moment.
(There are other ways the modification masses could be recovered, like the UNIMOD accessions i think.)

[colin-combe]

so, in PR #84
i think validation works.

It checks that the Seq element is there for target proteins.

I temporarily disabled a check that crosslinks have valid link sites because I think the main file @aozalevsky is using for testing does have invalid link sites in it.