Fairmat 2024: proposal on electron microscopy (EM)

[lukaspie]

Essence of changes

• Refactored version for NXem including feedback from the EM community
• Refactoring of NXimage, NXspectrum base classes and examples for their usage
• Examples for documenting method-specific results with some contextualization for processing EBSD as an example for diffraction, EDS for spectroscopy, as well as images
• More complete representation of microscope components
• Connection to all concepts within EMglossary community-agreed upon terminology set where it fits
• Backed by example data converted to NeXus files using NXem for at least one example for each microscope manufacturer and thousands of cases for EBSD as a specific technique
• Removal of reserved suffix _set within the name of base classes

Dependencies of this PR on other PRs
This PR depends heavily on others that make changes to existing base classes or introduce base classes that are to be used outside of APM as well. Thus, it will be much smaller once the discussion on these PRs is done and this particular branch here has been rebased:
#1410 (not used here anyway)
#1412 (but concepts not used in EM)
#1415
#1407 -> merged
#1408 -> merged
#1413 -> merged
#1414 -> merged
#1419 -> merged
#1420 -> merged
#1486 -> merged

EM-specific base classes
There are several new base classes that are currently only used in NXem, they are ordered in a sequence of dependencies to assure that reviewing adds building blocks gradually:

Conceptual modeling of aberrations that are relevant to assure the microscope is calibrated well enough to achive precision and accuracy and sharp images:
NXaberration
NXcorrector_cs

Base classes for talking about thermodynamic phases and crystal structure as used in NXem_ebsd
NXphase
NXunit_cell

Base classes for concepts and/or components of an EM microscope
NXebeam_column
NXevent_data_em
NXibeam_column
NXmanipulator (essentially the version from #1424)
NXoptical_system_em
NXscanbox_em

Specialized NXprocess instances for modularized reporting of parameter, configs, and methods for common imaging modalities and types of NXdata collected, postprocessing.
NXem_ebsd (that is the most detailed example given the large number of example data in the public domain)
NXem_eds
NXem_eels
NXem_img

Specializations of NXdata to not let NXem grow even larger but allow to reduce the number of possible combinations how images and spectra will be named to work towards a more homogeneous presentation of n-dimensional hyperslab data:
NXimage, NXspectrum

The appdef itself:
NXem, the application definition that uses currently all these concepts because of which ought to be reviewed together

Several edits were necessary in classes unrelated to NXem only for the purpose of making the docs compilable (these are e.g. NXapm, edits in domain-specific-structure rst files of the documentation, or tests that otherwise would have failed. With the changes being worked on in parallel to this PR, merging back the NIAC main (i.e. upstream/main from the perspective of the FAIRmat/nexus_definitions), ensure that these pragmatic edits will gradually become superseeded by the respective PRs.

Small base classes which in the development of NXem turned out as too insignificant/not meaty enough ones were reworked with their concepts merged e.g. in the NXem appdef:
- NXem_correlation -> removed, placed in NXem
- NXem_method -> removed, placed in NXem
- NXem_msr -> removed, placed in NXem
- NXem_sim -> removed, placed in NXem
- NXevent_data_em_set -> removed, placed in NXem
- NXinteraction_vol_em -> removed, placed in NXem

Changes for base classes within the realm of computational geometry
Are covered in #1421, #1529, their usage in NXem is planned and the places intentionally placed but commented out to disentangle these geometry-related PRs
NXcg_{point,triangle,*} classes

Likewise, changes for base classes within the realm of documentation concepts for representing the geometry and properties of crystal defects and microstructures to bridge to our friends in materials engineering including appdefs for the simulation of microstructure evolution are planned to become usable in NXem but also those where here intentionally placed but commented out to also disentangle this set of classes from those for EM:
NXmicrostructure_{*} -> previously named as NXms_* classes

Further context is provided via a list of dependant changes, see #1465

[mkuehbach]

Telco, NIAC request to change names of some base classes:

• NXinteraction_volume_em > NXem_interaction_volume
• NXlens_em > NXmagnetic_lens
• NXoptical_system_em > NXem_optical_system
• NXscanbox_em > NXscan_controller

[lukaspie]

Telco, NIAC request to change names of some base classes:

• NXlens_em > NXmagnetic_lens

I suggest to use NXelectromagnetic_lens, so that electrostatic lenses in XPS are also covered.

[mkuehbach]

@lukaspie @rettigl FYI - This PR will rename the base class NXlens_em to NXelectromagnetic_lens, might have practical consequences for datasets that you have already processed for NOMAD

[mkuehbach]

Is complete, @phyy-nx voting can start

[mkuehbach]

Hi Markus, NX_WAVENUMBER as units for the diffraction axis makes sense in theory, but oftentimes the calibration isn't right or even known to convert the pixel coordinates of the diffraction detector to reciprocal space coordinates. Instead, we might want to store the diffraction data in pixels and rely on the NXdetector and NXbeam groups to calibrate and convert from pixels to reciprocal space coordinates. Is this allowed/feasible in NXem?

Addressed via the last commit 31704f0

[mkuehbach]

@phyy-nx another signature of support for NXem (found the paper again while cleaning up on my system):

With that said, we have identified standards that capture
the needs of the materials electron microscopy community.
For bibliographic information, the Dublin Core can provide
answers to who, what, where, and when. For microscopy con-
ditions we have found HMSA, now an ISO standard (ISO/DIS
5820 under development),20 and the NXem extension to the
NeXus file format provide an effective ontology for describing
materials EM instrument conditions.18

found here https://doi.org/10.1557/s43577-025-00882-2

[phyy-nx]

Sweet find @mkuehbach!

[phyy-nx]

Hi all, this PR can now be moved to a vote. Please votes using emoji's, 👍 for yes, 👎 for no, anything else e.g. 👀 for abstain. Voting will close in 2 weeks!

[rayosborn]

There are a few files that I believe still don't follow the naming convention for domain-specific base classes, i.e., NXcorrector_cs, NXevent_data_em, and NXinstrument_em. I also note that NXaberration is documented as specific to the 'em' domain - should it be renamed NXem_aberration or do we think it is applicable as is to other uses of aberration corrections?

[mkuehbach]

Thank you @rayosborn comments:
I have implemented a solution via the follow-up proposal: FAIRmat-NFDI#402

We should not change this in a PR that is currently in a voting process, a few thoughts on this for each suggestion follows, as the changes here asked for are not only on docstrings

• NXevent_data_em and NXevent_data_apm, before the voting for NXem started NXapm was already accepted with here the naming convention accepted NXevent_data_apm, so NXevent_data_em is consistent with this but not - agree with you Ray - consistent with others now de facto apply naming conventions. I am supportive of contributing our part to resolve inconsistencies. In response I followed now the pattern NXmethod_classname so NXem_event_data and NXapm_event_data, this is implemented in the here mentioned PR 402
  This renames NXevent_data_{apm,em}, NXinstrument_{em,apm}
• NXaberration, the exemplar literature snippets detail for electron microscopy, the class itself though is not limited to this research field only, a sentence was added to the top-level doc string to make this also clear.
• NXcorrector_cs as mentioned for NXaberration also here the docstring was modified to clarify that the class is not only applicable in electron microscopy, given that we have NXmonochromator, NXaperture, I do not suggest to refactor to NXem_corrector_cs. Also these two edits, surplus dependent renamings in individual documents mentioning these classes were also edited in 402.

@phyy-nx I suggest to make a follow up PR with the content of proposal: FAIRmat-NFDI#402 after the votes on this EM PR here have passed. @rayosborn is this a compromise you support?

I have implemented a solution via the follow-up proposal:

There are a few files that I believe still don't follow the naming convention for domain-specific base classes, i.e., NXcorrector_cs, NXevent_data_em, and NXinstrument_em. I also note that NXaberration is documented as specific to the 'em' domain - should it be renamed NXem_aberration or do we think it is applicable as is to other uses of aberration corrections?

[phyy-nx]

Vote has passed. Thanks everyone!!

[sanbrock]

agreement from Telco: we can merge this PR, and then make a new PR from FAIRmat-NFDI#402 and then merge that one, too.

[mkuehbach]

Last comment addressed in #1585