ESMValTool: treatment of missing values

[axel-lauer]

I noticed that missing values are treated differently in the ESMValTool REF installation and the stand-alone ESMValTool installation. In case of the REF installation, one single missing value will lead to a missing result when calculating e.g. area means across many grid cells. This is different from (my) stand-alone ESMValTool installation, where missing values are ignored in such cases.

This could potentially be related to the default behavior of some Python packages used in the ESMValTool installation, but I do not know how to pin this down.

Here is an example, the left plot show the results of a time series averaged over a region with (my) stand-alone ESMValTool installation, the right plot the results obtained with the REF installation of ESMValTool. Note the gaps in the time series marked with the red arrows. This is potentially quite concerning.

[lewisjared]

The ESMValTool runs are performed in a pinned set of conda dependencies (). Can you try reproduce this by using the REF-generated miniconda environment to run your ESMValTool recipe directly? (location is determined via: https://climate-ref.readthedocs.io/en/latest/configuration/#paths_software).

That hopefully will bisect this to be a package or config error vs something with the REF-post-processing. We might need to pin some additional packages or include some additional ESMValTool config.

A diff of your local installation vs https://github.com/Climate-REF/climate-ref/blob/main/packages/climate-ref-esmvaltool/src/climate_ref_esmvaltool/requirements/conda-lock.yml may also help narrow things down.

[bouweandela]

The REF uses its own installation of ESMValTool, maybe the differences are related to that? You can see where it is installed with the command ref providers list, e.g. for me this is in ~/climate_ref/software/conda/esmvaltool-1b3bf252de04f119f83bfe65fac5aa040d9d97b4. To make sure that ESMValTool uses the right data when run by the REF, the REF writes its own configuration for ESMValTool and it's own recipe for every run. These are located in the scratch directory that is shown when you run the command ref config list.

To run ESMValTool exactly as the REF would, you can activate the ESMValTool conda environment used by the REF with the command conda activate ~/climate_ref/software/conda/esmvaltool-1b3bf252de04f119f83bfe65fac5aa040d9d97b4, find the output directory and go to it, e.g. picking the global-mean-timeseries diagnostic as an example, for me that would be cd ~/climate_ref/scratch/esmvaltool/global-mean-timeseries/ff49a69999a3c925212886831ddcd2f044fe6912/ (the last subdirectory is is named with a number representative of the input datasets used, so you'll see different subdirectories here for different input datasets), and run esmvaltool run --config-dir config recipe.yml. You may need to use a separate default configuration directory for ESMValTool by running e.g. export ESMVALTOOL_CONFIG_DIR=$HOME/esmvaltool_config_for_ref, to avoid picking up your standard one which may contain settings that you do not want to use with the REF. This can be useful to make sure ESMValTool only uses the input data defined by the REF and configure Dask to use the distributed scheduler with the REF, as that makes e.g. the cloud scatterplot diagnostics run much faster.