Skip to content

Conversation

@rayosborn
Copy link
Contributor

@rayosborn rayosborn commented Apr 30, 2025

A NIAC Telco discussion suggested that it was common practice to store the results of data analysis within the NXprocess group describing the operation. On reflection, I believe that this is a very valuable way of storing processed data, although I haven't used NXprocess groups in this way before. There are indeed many advantages to encapsulating both the description of the process (program name, data, input parameters) with the resulting data.

This PR does not change anything substantively. It modifies the base class documentation to make this usage more evident to non-specialists and explicity adds NXparameter and NXdata groups to store input parameters and the resulting data. Strictly speaking, both these group additions are redundant because their base classes have been added to the NXobject base class and are therefore already allowed. However, I think it is useful to add them here to show typical ways in which NXprocess groups are structured. As with groups in all base classes, their actual inclusion in a NeXus file is optional, unless an application definition explicitly requires them.

Copy link
Contributor

@mkuehbach mkuehbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the initiative lgtm, one important suggestion that can improve this pr

@PeterC-DLS
Copy link
Contributor

Can you comment on how this relates to https://manual.nexusformat.org/applying-nexus.html#processed-data? It seems to me that the original usage was to place one or more NXprocess groups in an NXentry to document how the processed data in that entry was derived. This is how it was used in Dawn's processing (video is here) to write the output NeXus files.

@sanbrock
Copy link
Contributor

sanbrock commented Jun 6, 2025

In fact, the prescrition of having separate entry foreach data processing step also suggests that if you want to specify restrictions/requirement on those steps, one must create seoarate Application definition for each of them.
While this can be useful, one can also imagine the use case where a single Application definition prescribes multiple data processing steps. Hence, I suggest to clarify in the documentation that multiple processing steps can be organised in multiple entries, or sequenced in a single entry.

@PeterC-DLS
Copy link
Contributor

PeterC-DLS commented Jun 6, 2025

Multiple processing steps must have a separate entry each.

Seems overly restrictive and should be relaxed!

Copy link
Contributor

@mkuehbach mkuehbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am okay with this PR getting merged before #1422 but then this branch needs to be updated with main first.

@mkuehbach
Copy link
Contributor

telco today @phyy-nx: @rayosborn please edit this paragraph and add to this PR:
Also in elsewhere in the manual

@rayosborn
Copy link
Contributor Author

rayosborn commented Jun 16, 2025

In the above commit, I have suggested changes to the manual, which better reflect the ways in which the NXprocess group could be used. One thing that we didn't discuss just now in the Telco is how the NXdata group produced by the NXprocess is identified if it is not stored within the NXprocess group. There is no field or link within the NXprocess group that specifies which NXdata group was generated. In the NXtomoproc application definition, the only way to tell that the NXdata group is the result of the action described in the NXprocess group is in the tag descriptions, which are, of course, not stored in the file itself. In this case, there is only one NXdata group, but there could be more.

I don't want to delay this PR any further, but I think in the future, we should consider making a recommendation that, if an application definition wants the NXdata group to be outside the NXprocess group, there should either be a link to the NXdata group within the NXprocess group (or vice versa) or that the path to the NXdata group should be stored as a field in the NXprocess group.

Copy link
Contributor

@mkuehbach mkuehbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase again nexusformat/definitions.git main branch. Other than this, approved from my side.

Copy link
Contributor

@PeterC-DLS PeterC-DLS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mkuehbach
Copy link
Contributor

mkuehbach commented Jun 18, 2025

@rayosborn this PR is ready, @phyy-nx to be voted upon. Needs to be rebased against main before merge. Vote can start like discussed at the last telco, including #1422

@phyy-nx
Copy link
Contributor

phyy-nx commented Jun 18, 2025

Hi folks, as discussed on the Telco, this PR can now me moved to an online vote. Please vote using emojis (👍 for yes, 👎 for now, anything else e.g. 👀 for abstain). Voting will close in 2 weeks. Thanks!

@rayosborn
Copy link
Contributor Author

@PeterC-DLS, you suggested the possibility of adding some text about adding a "default" attribute to the NeXus manual text, but I decided in the end that it made more sense to add the attribute explicitly to the NXprocess group itself. Please could you confirm that my change to the PR is syntactically correct as well as appropriate. There have already been a few votes concerning this PR, so I will revert the change if anyone thinks it has been added too late.

@phyy-nx
Copy link
Contributor

phyy-nx commented Jul 6, 2025

The vote has passed, thank you. @mkuehbach has requested a rebase, then it can be merged

rayosborn added 5 commits July 7, 2025 10:26
These are already allowed through extension of the NXobject group, but it is helpful to show that they have a particular purpose within this base class.
In principle, there could be more than one.
@rayosborn rayosborn force-pushed the update-nxprocess-documentation branch from 5b4fb4c to 75647df Compare July 7, 2025 15:27
@rayosborn
Copy link
Contributor Author

The rebase has been completed.

Copy link
Contributor

@mkuehbach mkuehbach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comments were addressed appropriately as discussed at 2025/07/07 telco

@rayosborn rayosborn merged commit 5fe30ba into nexusformat:main Jul 7, 2025
2 checks passed
@PeterC-DLS PeterC-DLS added this to the NXDL 2025 milestone Aug 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants