Ingest suite to download raw obs and convert it to ioda format, and ingest it to R2D2

[mer-a-o]

Description

This PR adds an end-to-end "download-convert-ingest" pipeline with initial support for OMPS O3 Nadir Mapper (OMPS-O3-NM) data in the GEOS-CF interface.

Summary of updates

• DownloadObs: downloads raw observation files from remote servers (e.g. via wget) for a given cycle window. Authenticates using ~/.netrc instead of a hardcoded Earthdata token.

• ConvertObsToIoda: runs an ioda-converters script for the downloaded raw files to produce a single IODA-formatted NetCDF per cycle. The converter binary directory is configurable via converter_path, falling back to jedi_bundle/build/bin.

• flow.cylc in obs ingest suite now supports optional DownloadObs -> ConvertObsToIoda -> IngestObs task chain, gated by a new download_convert_pipeline boolean flag. When the flag is false, the suite runs IngestObs alone (preserving existing behavior).

  • Suggestions for a better approach are welcome.

• In IngestObs, a local acquisition method is added so IngestObs can pick up files produced by ConvertObsToIoda within the same cycle, in addition to the existing cp (and s3) path methods.

• New obs configuration files (geos_cf)

  • download_observations/omps_o3_nm.yaml: wget pipeline config with filename pattern and date placeholders.
  • convert_observations/omps_o3_nm.yaml: ioda-converter script name, output filename template, and extra converter flags (-q, -e).
  • ingest_observations/omps_o3_nm.yaml: R2D2 ingest config using the new local retrieval method.

Dependencies

• update ompso3 for nasa gmao JCSDA-internal/ioda-converters#1768

closes: #769

[mranst]

I was able to run the default case 👍

[jeromebarre]

Looks good to me now!

[mranst]

One fix needs to be made to the suite_config but otherwise looks good

[mer-a-o]

@mranst This PR is ready for merge (if all your comments are answered).