convert_bufr update: bufr-query bufr2netcdf.x

[cohen-seth]

Description

Updated 05-06-2026:

Log edits and error handling cleanup

Ouptut ioda files now have sat_name and satid in the file name (e.g., cosmic2_755)

File Cleanup: convert_bufr will now remove the leftover 'temporary*.nc' files that don't get renamed due to an existing converted file for that obstype date

To Do:

1. Add override option to overwrite existing files
2. Make edits that use swell utilities
   • clean_cycle
3. Only convert files for obs types in the experiment yaml. Currently tries to convert all the bufr files it finds in the cylce directory

---

Updating the bufr_to_ioda.py task from the convert_bufr suite to use the new executable bufr2netcdf.x from bufr-query. The previous executable bufr2ioda.x is deprecated. bufr2netcdf.x and bufr2ioda.x each require a specific yaml file format so a new yaml directory was added for the bufr2netcdf.x specific yamls for all SPOC yamls (https://github.com/NOAA-EMC/spoc/tree/develop/dump/config/atmosphere).

Produces one file for each satid ('{splits/satd}' method).

The key changes to bufr_to_ioda.py were:

• removing steps for generating the mapping yamls (the older method changed the actual contents of the yaml files)
• changes to account for the (full) CLI command changing from bufr2ioda.x [bufr file] [bufr_mapping.yaml] to bufr2netcdf.x [bufr file] [bufr_mapping.yaml]
• added observations type to the defaults in the convert_bufr/suite_config.py and question defaults
• bufr_to_ioda.py executable changed from bufr2ioda.x (deprecate) to bufr2netcdf.x
• New yaml files with format specific to bufr2netcdf.x added to new directory in configuration/jedi/bufr2ioda/bufr2netcdf_x/
• Working for multiple observation types

Impact

Should make the next set of changes easier if/when we integrate the use of the bufr-query ObsBuilder

Testing

Swell SPOC yaml testing convert_bufr

yaml	tested	working	questions_default	compared against other nc4 outside swell	Notes
spoc_prepbufr_aircraft.yaml	🛑	TBD			Requires the custom naming structure check (acft_profiles).
spoc_radiance_1bamua.yaml	✅	✅	*
spoc_radiance_1bmhs.yaml	✅	✅	*
spoc_radiance_amsr2.yaml
spoc_radiance_amsua_1bamua.yaml	🛑
spoc_radiance_amsua_esamua.yaml
spoc_radiance_atms.yaml	✅	✅	*
spoc_radiance_avcsam.yaml	✅	✅	*
spoc_radiance_avcspm.yaml	✅	✅	*
spoc_radiance_avhrr.yaml	✅	🛑✅			Used for BOTH avcsam/avcspm conversions.
spoc_radiance_cris-fsr.yaml	🛑
spoc_radiance_gmi.yaml
spoc_radiance_gpsro.yaml	✅	✅	*		Ran successfully in previous log output (created files for metop, cosmic2, planetiq, etc.).
spoc_radiance_iasi.yaml	🛑				Note: Dict has both spoc_radiance_mtiasi.yaml and spoc_radiance_iasi.yaml mapped to mtiasi obs.
spoc_radiance_mhs_1bmhs.yaml	🛑
spoc_radiance_mtiasi.yaml	✅	✅
spoc_radiance_ssmis.yaml	✅	✅
spoc_retrieval_ozone_omi.yaml
spoc_retrieval_ozone_ompslp.yaml
spoc_retrieval_ozone_ompstc.yaml

[mranst]

Nice, I was able to run the conversion successfully. I would have a look at some of the try-catch logic in BufrToIoda, I think you're going to end up passing some issues that should really cause an error, like not being able to find the jedi executable.

[mranst]

FYI @ftgoktas this PR changes how the BufrToIoda task works - I think you mentioned you were using it

[mranst]

A few notes, otherwise I think it's ready