SNAP-based Coregistration

[Simon-van-Diepen]

Adds:

• snap_preparation job: prepares the run folder and the XML graphs, which are numbered for jobarray execution

• snap_run job: an arrayjob where each subjob runs one graph, and outputs a .znap.zip archive

• snap_cleanup job: unzips all the .znap.zip archives into .znap, which are readable using Python's zarr package, and removes the .znap.zip archives

• support for arrayjobs within Caroline (in scheduler, preparation, and email)

• snap-coregistration dependency

[Simon-van-Diepen]

@fnattino I requested your review for two specific files (the rest you can ignore if you want):

• templates/snap/generate-snap-graphs.sh
• templates/snap/run-snap-graph.sh

These two files contain the calls to snap-run and gpt. I would like to verify with you that the way this is being called is the correct and optimal approach.

I have generated results for my testing area of interest, they are located in /project/caroline/Share/users/caroline-admin/caroline-test/test_nl_amsterdam_v3/stack-snap/nl_amsterdam_s1_dsc_t037 if you're interested (it's a 1x1 km^2 area over Amsterdam for half a year so it's nice and fast). Each date has its own ZNAP archive, readable via the zarr package in Python.

@rogerkuou @FreekvanLeijen FYI

[fnattino]

Looks good @Simon-van-Diepen ! Nice that this seems to work, I have added just a couple of comments.

And just to confirm, have you added the following lines to your ~/.esa_snap/etc/snap.properties?

znap.compression.level=5
znap.compressor.id=zlib
znap.use.zip.archive=false

This is to have GPT already writing unzipped ZNAP dirs. I see you have a note on this in the PR description, but I could not see any clean up script, so I imagine you already included this.

Related to this, maybe it would be good to point GPT to a common config files, saved somewhere in the /project/caroline/Software ? This way everyone running caroline would use the same settings without having to modify a configuration file in their HOME folder. Let me know what you think, I can look up what exactly needs to be changed in the GPT settings.

[Simon-van-Diepen]

@fnattino regarding your general comment, I have indeed not added those three lines to my config file. The two main reasons for that are that I want CAROLINE to be able to completely install itself without having to modify settings like that, and that this is a global config file meaning the testing environment and live environment are no longer fully separated (so a test with different settings would also impact the live runs immediately). Hence my workaround with an unzipping script (which is hidden in preparation.py), which also immediately changes the access so that it is accessible to everyone.

However, your idea of a CAROLINE SNAP config file is interesting, if this could be broadcast directly to the GPT command, since then multiple versions of that config file can exist concurrently without interfering. Do you know how to do that?

[fnattino]

@fnattino regarding your general comment, I have indeed not added those three lines to my config file. The two main reasons for that are that I want CAROLINE to be able to completely install itself without having to modify settings like that, and that this is a global config file meaning the testing environment and live environment are no longer fully separated (so a test with different settings would also impact the live runs immediately). Hence my workaround with an unzipping script (which is hidden in preparation.py), which also immediately changes the access so that it is accessible to everyone.

I see, and indeed it would be nice not to have dependencies on other config files. Unfortunately, GPT does not allow you to specify all config options as CLI arguments (it is not clear to me why).

However, your idea of a CAROLINE SNAP config file is interesting, if this could be broadcast directly to the GPT command, since then multiple versions of that config file can exist concurrently without interfering. Do you know how to do that?

Well, as far I can see you cannot specify the path to a config file as a GPT CLI argument. What I meant with my comment, is that you could add the Zarr configuration option to the global SNAP config file, i.e. /project/caroline/Software/snap/12.0.0/etc/snap.properties which will apply to any user using the shared SNAP installation. Note that some options in that file are already customized for usage on Spider (e.g. snap.userdir is set to ${HOME}/.esa_snap vs default ${HOME}/.snap, since .snap folders are reserved on the CephFS filesystem used on Spider), and I think Niels is maintaining the installation scripts.

[Simon-van-Diepen]

@fnattino since that'd be a global change I'll check with the group if it will break someone's implementation if there would be unzipped ZNAP files being outputted. If not I'll shoot Niels an email to ask to add this (I have the permissions to do so too but I think it's cleaner to go via him)

Thanks for all your feedback! I'm running one more fullscale test now to see if everything is still working as intended, and then I'll merge.

[Simon-van-Diepen]

@fnattino after discussion with the group those three lines have been added. Everything works as intended, so I'll merge now. Thanks for your feedback!