Speed up file copies in marine letkf and recentering

[shlyaeva]

gdas_marineanlletkf and gdas_marineanlecen take a long time to copy ensembles between directories (in some cases it's longer to copy files than to actually run an application that computes LETKF or recentering). This is a waste of resources, especially given that it's done in a multinode task.

There were some conversations about speeding this up by either parallel copies, or allowing links, but I don't know where that's at. I expect improving on this would significantly speed up these tasks.

[shlyaeva]

FYI @AndrewEichmann-NOAA, @guillaumevernieres.

@CatherineThomas-NOAA, @JessicaMeixner-NOAA: do we do something special for this in marine ensemble recentering for v17?
@TravisElless-NOAA any suggestions on the best way to speed this up?

[JessicaMeixner-NOAA]

There is the mpmd script that's used in other jobs to do parallel copying. here's an example of where/how that is used in the forecast: https://github.com/NOAA-EMC/global-workflow/blob/dev/gfs.v17/ush/forecast_postdet.sh#L418-L439

[AndrewEichmann-NOAA]

@JessicaMeixner-NOAA Can you see any objection to our using the same approach?

[JessicaMeixner-NOAA]

@TravisElless-NOAA and @DavidHuber-NOAA - would have a better idea, but I don't know why you would. They would also have the latest on the links, etc.