Use hyperspy?

[domna]

Hey @MarJMue and @litdotcom,

I searched through the web as I was thinking about modularising this project and to make it more generic and stumbled upon hyperspy, which solves exactly the problem we are facing here.
It has a stronger focus on electron spectroscopy techniques, but I think it's also applicable to optical spectroscopy.

They also have an ipywidgets module. So maybe it's better to contribute the cosmic removal algorithm and file importers there and focus on adding features there instead of creating yet another package.
What do you think?

[litdotcom]

Hey, it sounds like a good idea in general to merge with a bigger project. They deal with spectra of the same dimensions as we do (e.g. https://hyperspy.org/hyperspy-doc/v1.6/user_guide/visualisation.html) so the modules should fit into theirs. However I would suggest to create something that's running fine on it's own and then try to merge it into hyperspy. P.S.: Therefore I will need to invest some more time into it which I didn't manage yet. I might be able to after December 1st otherwise when I get back to Germany December 16th.

…

On Tue, 8 Nov 2022, 10:03 Florian Dobener, ***@***.***> wrote: Hey @MarJMue <https://github.com/MarJMue> and @litdotcom <https://github.com/litdotcom>, I searched through the web as I was thinking about modularising this project and to make it more generic and stumbled upon hyperspy <https://hyperspy.org>, which solves exactly the problem we are facing here. It has a stronger focus on electron spectroscopy techniques, but I think it's also applicable to optical spectroscopy. They also have an ipywidgets module. So maybe it's better to contribute the cosmic removal algorithm and file importers there and focus on adding features there instead of creating yet another package. What do you think? — Reply to this email directly, view it on GitHub <#8>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ5APURZDXFXK4JKTMDO4W3WHIJNPANCNFSM6AAAAAAR2CGD5I> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[domna]

Honestly, I think it is better to orient ourselves towards hyperspy right from the start. Merging this project as is into hyperspy is probably not possible due to a very different program design. Also hyperspy already introduces a lot of features I intended to implement here (e.g. registering of transformations and filters on the data).

There is already a lot out there which provides similar features to what we are searching for here:

• Lumispy is an extension for luminescence analysis within hyperspy
• hyperspy ipywidgets is an extension to display the hyperspy graphs inside jupyterlab

It could very well be that lumispy already has a feature for cosmic removal and if not I would suggest contributing it there.
The only thing missing is that hyperspy is not able to read the dataformat we use here. There we could go for two approaches: Convert the h5 or csv into a supported format of hyperspy (e.g. hspy, netCDF or nexus) or to contribute a loader to hyperspy. From my point of view the first option is the better one.

[litdotcom]

Yes, it seems better to orient directly towards hyperspy/lumispy. The extension lumispy doesn't offer a possibility to remove cosmical signatures yet (based on the information given in their documentation). Do you intend to shut this project down after merging the key features into hyperspy?

…

On Wed, 9 Nov 2022, 09:55 Florian Dobener, ***@***.***> wrote: Honestly, I think it is better to orient ourselves towards hyperspy right from the start. Merging this project as is into hyperspy is probably not possible due to a very different program design. Also hyperspy already introduces a lot of features I intended to implement here (e.g. registering of transformations and filters on the data). There is already a lot out there which provides similar features to what we are searching for here: - Lumispy <https://github.com/lumispy/lumispy> is an extension for luminescence analysis within hyperspy - hyperspy ipywidgets <https://github.com/hyperspy/hyperspy_gui_ipywidgets> is an extension to display the hyperspy graphs inside jupyterlab It could very well be that lumispy already has a feature for cosmic removal and if not I would suggest contributing it there. The only thing missing is that hyperspy is not able to read the dataformat we use here. There we could go for two approaches: Convert the h5 or csv into a supported format of hyperspy (e.g. hspy <https://hyperspy.org/hyperspy-doc/v1.6/user_guide/io.html?highlight=reading#hspy-format>, netCDF <https://hyperspy.org/hyperspy-doc/v1.6/user_guide/io.html?highlight=reading#netcdf> or nexus <https://hyperspy.org/hyperspy-doc/v1.6/user_guide/io.html?highlight=reading#nexus>) or to contribute a loader to hyperspy. From my point of view the first option is the better one. — Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ5APUWSK2EQWUGKP3XK4VTWHNRH5ANCNFSM6AAAAAAR2CGD5I> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[domna]

Yes, from what I see it seems to be an early project. I'm not sure if I'll shut down this project, I'll keep it at least open for people using it at JLU as long as the same analysis cannot be performed by hyperspy. For this I'll need to thoroughly check through one example and see how it works in hyperspy. Another thing I like in this project is the plotly based visualizations which I find really nice to work with and nice to look at, so there would be the question how this could be added to hyperspy, too.