Adding cycling to 3dvar_cf suite

[mer-a-o]

Description

This PR adds a new cycling 3DVar suite for GEOS-CF. The experiment is designed to run a 12h forecast starting from the beginning of the assimilation window and apply the increment from the middle of the window using IAU. The forecast length and frequency can be set using forecast_length and forecast_output_frequency but I didn't test changing the forecast length.

Summary of changes:

• RC files that get templated with SWELL are in src/swell/configuration/jedi/interfaces/geos_cf/namelists. The remaining RC files (static) are copied into the scratch directory from geos_cf_run_dir.

• get_background task fetches backgrounds from R2D2 using background_experiment in the config as the experiment ID. In cycling experiments (those with "cycle" in their name), the experiment ID is set to background_experiment for the first cycle. For subsequent cycles, it is set to the r2d2_experiment_id of the current experiment, so that backgrounds from the previous cycle are fetched for the variational task in the current one.

• save_forecast task stores forecasts in R2D2 using r2d2_experiment_id as the experiment ID. These files are then fetched from R2D2 during the get_background task.

• Restarts are saved (save_restart) and fetched (get_restart) from R2D2. To save space, restart files are not stored on R2D2 at every cycle. The rst_store_interval key controls how many cycles pass before restart files are stored as real files rather than symlinks. In intermediate cycles, restarts are saved as symlinks.

• prep_forecast prepares the scratch directory within the current cycle's run directory:

  • Copies static RC files from geos_cf_run_dir and copies/edits RC files from src/swell/configuration/jedi/interfaces/geos_cf/namelists.
  • Copies GEOS-FP files for replay into the scratch directory.
  • Changes the format of the JEDI increment file using inc_template.

• run_forecast submits gcm_run_geoscf.j (a lighter version of gcm_run.j used for running GEOS-CF) to the queue and waits until the job finishes. One limitation: the Cylc interface shows this job as "running" whether it is waiting in the queue or actually executing. There may be better approaches for handling gcm_run.j execution.

• clean_cycle.py now cleans the scratch directory of the previous cycle after the current cycle completes. This way, the restart files from the previous cycle that are saved as symlinks are kept until the current cycle uses them.

[rtodling]

this file should not exist in the repo - it should only exist in the experiment. A way to avoid this file being in the repo is to have swell create this file based on the initial date/time of the experiment. From there on, the experiment (GCM) will create this file on its own; the file can then be recycled from one cycle to the next.

[mer-a-o]

removing cap_restart. Note that prep_forecast.py handles writing the correct initial date/time for the experiment.

[rtodling]

We should talk about the naming conventions for these templated frequencies and such.

[rtodling]

I am trying to understand the reason for the changes here. Is this because the backgrounds for various cycles for the swell experiment can be located outside of r2d2?

[mer-a-o]

Please see the comment in the code. For the first cycle the backgrounds are fetched from R2D2 using the experiment_id specified by the user. For the cycles after that swell ('get_background.py') fetches the backgrounds files from the 12h forecast done in the previous cycles. Those forecast files from the previous cycle get updloaded to R2D2 with the current experiment id. In the following cycle this experiment id is used to fetch those forecast files and will be used as backgrounds for the current cycle.

[rtodling]

I puzzled as to why this RC and other RCs and GEOS yaml need to be here. These files should be in the tag of GEOS that supports CF, no? What am I missing?

[mer-a-o]

For each cycle, FileTemplate is replaced to point to the JEDI increment file. I'm keeping all the RC files that are modified for each cycle in SWELL. The rest of RC files are copied from a tagged version of GEOS specified in suite_config.py: geos_cf_run_dir('/discover/nobackup/mabdiosk/rundir/GCv14.0_GCMv1.17_c90_Skylab')
I think this way we can easily keep track of what goes into the swell experiment.

[rtodling]

Same comment as above for RC files ...

[rtodling]

Same comment as for the RC files - this should be in the GEOS tag and in the settings of your experiment.

[jeromebarre]

Should this be moved out of the draft status at this point? Ideally we want this ready for merge ASAP even if this is isn't perfect this is IMO good enough and replicates what has been done in slylab.
There are fixes to be done on the coding norms and the swell CI. Which is something to clean in saveforecast.

Over time a few improvements on top of my head:

• getting met forcings for replay from the original locations and not from my dir in geos
• considering handling tarballs for restarts and not individual files
• GEOS-CF forecast logs written in the correct locations. But maybe this was fixed since last time I tested

[mranst]

Since this script and the RunForecast task are relatively simple, would it be possible to run the executable directly from the task runtime like how Doruk uses gcm_run.j?

    [[RunForecast-{{model_component}}]]
        script = "{{experiment_path}}/run/$datetime/{{model_component}}/gcm_run_geoscf.j"
        platform = {{platform}}
        execution time limit = {{scheduling["RunForecast"]["execution_time_limit"]}}
        [[[directives]]]
            --output="{{experiment_path}}/run/$datetime/{{model_component}}/gcm_run.log"
        {%- for key, value in scheduling["RunForecast"]["directives"][model_component].items() %}
            --{{key}} = {{value}}
        {%- endfor %}

[mranst]

If I'm not mistaken the RunForecast task will be queued into slurm, then this will allocate its own job on top of that. This command also failed for me when I tried it

[mranst]

Will this affect 3dvar_marine_cycle and 3dfgat_marine_cycle?