S2S post - issue with running a single model

[karsenau]

The below issues were discovered when trying to submit a single S2S post step for 1 model only -- THE S2S POST WEEKLY SCRIPTS FAIL AND USER IS UNABLE TO RESUBMIT S2SPOST WEEKLY JOB  VIA SBATCH IN THE SCRATCH DIRECTORY 

1) When running 1 model only, running the S2S post step by itself OR WHEN starting from the "POST" step and running through remainder steps (e.g., you have to patch a model s2spost output, like just CFSv2) – the below errors arise -- 
 
Traceback (most recent call last): 
  File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 344, in  
    Ghis2sProgram().main() 
  File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 341, in main 
    self.run() 
  File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 321, in run 
    self._run_s2s_workflow() 
  File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 183, in _run_s2s_workflow 
    s2s.submit_jobs() 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2s_app/s2s_run.py", line 751, in submit_jobs 
    prev_ids = get_previds(jfile) 
               ^^^^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2s_app/s2s_run.py", line 643, in get_previds 
    previd_list.append(self.schedule[file]['jobid']) 
                       ~~~~~~~~~~~~~^^^^^^ 
KeyError: 's2spost_02_run.j' 
·       Looks like it tries to submit another model, I.e., GEOSv2 here  
·       And gives the above error 
 
 
2). Similar to the bug above – s2spost_weekly fails if s2spost is run as an individual step for an individual model:   
 
Traceback (most recent call last): 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info 
    file = self._cache[self._key] 
           ~~~~~~~~~~~^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in getitem 
    value = self._cache[key] 
            ~~~~~~~~~~~^^^^^ 
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/discover/nobackup/projects/ghilis/karsenau/E2ES_7.7_CYLC_12012025/scratch/202510/s2spost/CFSv2/PS.557WW_SC.U_DI.C_GP.LIS-S2S-CFSV2_GR.C0P25DEG_AR.GLOBAL_PA.ALL_DD.20251001_DT.0000_FD.20251009_DT.0000_DF.NC',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'd1f92cca-8c47-4f2f-9098-caa5f00c8ab6'] 
 
During handling of the above exception, another exception occurred: 
 
Traceback (most recent call last): 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2spost//process_fcst_files.py", line 256, in  
    _driver() 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2spost//process_fcst_files.py", line 251, in _driver 
    proc_time_period(config, configfile, topdatadir, fcstdate, startdate, 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2spost//process_fcst_files.py", line 238, in proc_time_period 
    agg_driver(argv, logger, subtask) 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7/lis/utils/usaf/S2S/ghis2s/s2spost/temporal_aggregate.py", line 411, in agg_driver 
    outfile = create_time_aggregated_file_xarray( 
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7/lis/utils/usaf/S2S/ghis2s/s2spost/temporal_aggregate.py", line 240, in create_time_aggregated_file_xarray 
    ds_first = xr.open_dataset(daily_files[0]) 
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/api.py", line 686, in open_dataset 
    backend_ds = backend.open_dataset( 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/netCDF4.py", line 666, in open_dataset 
    store = NetCDF4DataStore.open( 
            ^^^^^^^^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/netCDF4.py", line 452, in open 
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) 
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/netCDF4.py", line 393, in init 
    self.format = self.ds.data_model 
                  ^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/netCDF4.py", line 461, in ds 
    return self.acquire() 
           ^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/netCDF4.py", line 455, in _acquire 
    with self._manager.acquire_context(needs_lock) as root: 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/contextlib.py", line 137, in enter 
    return next(self.gen) 
           ^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context 
    file, cached = self._acquire_with_cache_info(needs_lock) 
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
  File "/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info 
    file = self._opener(*self._args, **kwargs) 
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
  File "src/netCDF4/_netCDF4.pyx", line 2540, in netCDF4._netCDF4.Dataset.init 
  File "src/netCDF4/_netCDF4.pyx", line 2055, in netCDF4._netCDF4._get_vars 
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success 
RuntimeError: NetCDF: Can't open HDF5 attribute 

Targeted for LISV7.8

@smahanam @jvgeiger

[smahanam]

Need a test case to replicate your problem. Basically, I will create a link to your lis_fcst directory with single NMME model and try to run this s2spost.

[smahanam]

I need some other information too:
s2s_run.py -y year -m month -c config

few examples for 2 separate cases:

1. NMME CFSv2 files are fine
2. NMME CFSv2 files have undefined slices

[karsenau]

Here is one of the cases that I ran into this issue (I was rerunning the past few month initial forecasts) - Oct-2025 forecast, CFSv2 only:

/discover/nobackup/projects/ghilis/karsenau/E2ES_7.7_CYLC_12012025/scratch/202510/s2spost/logs/s2spost_weekly_01_54600946.err

I was running with this script:
test_ghis2s_onestep_cfsv2-only.sh

From here:
/discover/nobackup/projects/ghilis/karsenau/E2ES_7.7_CYLC_12012025

And the associated s2s_config file (for CFSv2 only):
s2s_config_global_fcast_cfsv2

[smahanam]

How about the NMME CFSv2 missing slices case

[karsenau]

That is a separate issue

[smahanam]

Then, create a separate issue and post few examples

[karsenau]

yes, separate ticket exists for that

[smahanam]

/discover/nobackup/projects/ghilis/smahanam/bug_fixes/

I ran s2spost with CFSv2 alone, it just ran.
I used SLURM, you have used Cylc, let me rerun.

[karsenau]

Ok, thanks!

Is that with any code updates? Which version of code - 7.7.1 or master?

[smahanam]

No ! I just couldn't replicate your problem.

[karsenau]

You need to check out the latest master branch and run that. Not with any other branch at this time.

[smahanam]

I understand your point. I am running from my sandbox. I haven't modified s2spost, s2smetrics, and s2splots after 7.7
HydroSFS changes are in BCSD
Domain decomposition updates are in LISDA, LDTICS, and LIS_FCST
I mean s2spost, s2smetrics, and s2splots are as same as 7.7

[smahanam]

Can you start fresh
mkdir bug_fixes
cd bug_fixes
/bin/ln -s /discover/nobackup/projects/ghilis/karsenau/E2ES_7.7_CYLC_12012025/lis_fcst/
Copy s2s_config_global_fcst and set NMME models to just CFSv2
python $PYTHONPATH/ghis2s/s2s_app/s2s_run.py -y 2025 -m 10 -c s2s_config_global_fcast -s POST -o -j

For me, it just ran

[karsenau]

Ok, thanks for those outlined steps that worked for you.
It does work from that approach - initiating the commands+jobs from the command line.

I had been running the same setup but via our "test_ghis2s.sh" script -- that is where the problem was being run into.

That is where I'm seeing this error:

sh test_ghis2s_onestep_cfsv2-only.sh

/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/

2026-03-24 22:39:31,214 - INFO - Starting ghis2s S2S workflow integration
/discover/nobackup/projects/usaf_lis/emkemp/miniforge3/20250304_py311_sles15_hpc11/lib/python3.11/site-packages/esmpy/interface/loadESMF.py:94: VersionWarning: ESMF installation version 8.8.0, ESMPy version 8.8.0b0
warnings.warn("ESMF installation version {}, ESMPy version {}".format(
2026-03-24 22:39:55,976 - INFO - Creating working directories and job files for S2S forecast initialized on 2025-10-01
October ...
Symbolic link lis_fcst already exists, skipping creation.
Symbolic link input already exists, skipping creation.
Symbolic link CFSv2 already exists, skipping creation.
[INFO] Submitting batch job for cf_CFSv2_202510
[INFO] Submitting batch job for cf_CFSv2_202511
[INFO] Submitting batch job for cf_CFSv2_202512
[INFO] Submitting batch job for cf_CFSv2_202601
[INFO] Submitting batch job for cf_CFSv2_202602
[INFO] Submitting batch job for cf_CFSv2_202603
[INFO] Submitting batch job for cf_CFSv2_202604
[INFO] Submitting batch job for cf_CFSv2_202605
[INFO] Submitting batch job for cf_CFSv2_202606
Submitted successfully. Job ID: 54902273; Job Script: s2spost_01_run.j
Traceback (most recent call last):
File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 344, in
Ghis2sProgram().main()
File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 341, in main
self.run()
File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 321, in run
self._run_s2s_workflow()
File "/gpfsm/dnb06/projects/p204/karsenau/GHI-repos/ghi-apps/bin/ghis2s_program.py", line 183, in _run_s2s_workflow
s2s.submit_jobs()
File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2s_app/s2s_run.py", line 751, in submit_jobs
prev_ids = get_previds(jfile)
^^^^^^^^^^^^^^^^^^
File "/discover/nobackup/projects/ghilis/karsenau/LISF_7.7//lis/utils/usaf/S2S/ghis2s/s2s_app/s2s_run.py", line 643, in get_previds
previd_list.append(self.schedule[file]['jobid'])
~~~~~~~~~~~~~^^^^^^
KeyError: 's2spost_02_run.j'