Handling CMORCheckErrors

[prosku]

Dear ESMValTool team,
first of all, thanks for providing this impressive tool, I've had a fun time diving into it so far!

The project I am working on requires loading AMIP simulations from as many models and model versions for as many variables as possible.
Hence I'm specifying as many models as possible in my dataset request within my recipe:

        - project: CMIP6
          exp: amip
          dataset: '*'
          institute: [CSIRO-ARCCSS, CSIRO, BCC, CAMS, NCAR, CMCC, CCCma, CAS, FIO-QLNM, MPI-M, CCCR-IITM, MIROC, HAMMOZ-Consortium, MRI, NUIST, NCC, SNU, AS-RCEC] 

A few examples of issues I am facing with corresponding questions on how to handle them:

1. When I add NASA-GISS to the institute list, I get a CMORCheckError:

esmvalcore.cmor.check.CMORCheckError: There were errors in variable ts:
 Time coordinate for var time is not monotonic
 time: Frequency mon does not match input data

Investigating the file in question (/work/bd0854/DATA/ESMValTool2/CMIP6_DKRZ/CMIP/NASA-GISS/GISS-E2-1-G/amip/r2i1p3f1/Amon/ts/gn/v20190522/ts_Amon_GISS-E2-1-G_amip_r2i1p3f1_gn_200101-201412.nc), I indeed see that the last months have 0 values (inside the ts values but also for the time dimension). Since the data is lost after 2007, this simulation is actually worthless for my analysis.
Hence the question:

• Is there a way to tell the recipe to skip datasets that throw CMOR Errors? That would be the ideal solution for me, since the recipe runs over many variables, and the datasets to exclude could be different for each one of them and it would be great to not have to specify which to exclude.
  or else:
• Is there a way to specify datasets to be skipped? Since I'd like to include all datasets possible, it would be easier for me to specify to exclude those for which I notice errors rather than to specify all that should be included.

2. Adding MOHC to the institute list, there's a KeyError:

  File "/sw/spack-levante/mambaforge-23.11.0-0-Linux-x86_64-befbel/lib/python3.10/site-packages/esmvalcore/cmor/fix.py", line 195, in fix_metadata
    cube_list = fix.fix_metadata(cube_list)
  File "/sw/spack-levante/mambaforge-23.11.0-0-Linux-x86_64-befbel/lib/python3.10/site-packages/esmvalcore/cmor/_fixes/cmip6/ukesm1_0_ll.py", line 26, in fix_metadata
    if cube.attributes[parent_units] == bad_value:
  File "/sw/spack-levante/mambaforge-23.11.0-0-Linux-x86_64-befbel/lib/python3.10/site-packages/iris/cube.py", line 1021, in __getitem__
    return store[key]
KeyError: 'parent_time_units'

Looking at those files I figured that the KeyError is missing from the exception:

                if cube.attributes[parent_units] == bad_value:
                    cube.attributes[parent_units] = 'days since 1850-01-01'
            except AttributeError:
                pass

Now one approach to handle this would be to skip the particular model version, as I'm asking in point 1. The other, as I've gathered from #3160, is to follow the developer instructions (https://docs.esmvaltool.org/projects/ESMValCore/en/latest/develop/fixing_data.html#fix-structure) to download my own copy of ESMValCore, test whether my suspicion is correct, and contribute a fix. Is that correct?
The contributor guidelines anyways ask to "please discuss your idea with the development team before getting started", so I thought I'd first ask in this way whether this would indeed be the path to follow.

3. I'll ask another question related to point 1 here: as I expand to more variables, some of them are not present in all the datasets that I specify in the recipe (at least for some years), which throws an error (in the recipe I'm attaching, mrro is the variable not present in all datasets) . As in 1., ideally I'd like to continue with the recipe just ignoring those datasets and going ahead for all other ones. Is there a way to do that?

recipe_collect-means_share-github.txt

I hope my questions are clear, otherwise please request more information. Thanks in advance for your help!

Ulrike

[jlenh]

Hi Ulrike,
Glad to see you pick up ESMValTool after the workshop 🙂

To try to answer your questions (I might need someone else from the @ESMValGroup/technical-lead-development-team to chime in):

• I do not know of any way to skip datasets when requesting all possible values for a facet in a recipe. In this case it would be probably too complicated to differentiate between actual errors and just skipping unfitting models. As you mentioned, you could relax the CMOR checks errors if you are still interested in some of these models (cf. last point here too).
• The easiest way to handle different variables would be to have a separate recipe for each variable since you run only preprocessing on the variables right? You could then figure out which datasets are valid for each variable and run the recipes without running into too much trouble (a bit cumbersome I agree).
• If you encounter small CMOR errors that should not bother you too much in your computations then you can indeed relax the level of the CMOR checks or, if needed, implement a fix for the specific model and variable that gives you trouble.

Hope that helps, maybe some wiser people like @valeriupredoi or @bouweandela would have more detailed and complete answers to your questions 🤓

[prosku]

Hi Julien,
thanks for getting back to me!

So regarding CMOR errors I can either relax the checks, exclude the models from the analysis manually (by switching to a list of models that I do want included and deleting the ones that give trouble or by specifying exclusion if someone else knows how this is possible) or fix them - that makes sense.

Regarding the variables: I do run diagnostics on the variables as well - including one that plots the output from all variables together in the end of the script. Thus, if I switch to one recipe per variable, that will mean redundancies between the recipes and including all the different recipes again in something larger like a Makefile that takes care of the summary plot. That seems cumbersome indeed, and also like a workflow that ESMValTool appears to be made to avoid. Would it instead for example be possible to define one dataset for each variable (with the models that work for it), and then specify that dataset in the diagnostics for each variable?

Thanks for your help!

[jlenh]

A way to go to for having the datasets defined for each variable but also not duplicated everywhere in the recipe would be to create anchors/pointers and use these through the additional_datasets facet for each variable. This allows you to specify the desired datasets for each variable separately while only defining once the datasets in the recipe.

Some examples can be found in the tutorial here (bonus exercises section), in this recipe here, or in this one here.

In your case, your recipe could look something like:

...

timerange_amip: &timerange_amip
    timerange: '1979/2008'

areacella_suppl: &areacella_suppl
    - {short_name: areacella, mip: fx, exp: '*', activity: '*', ensemble: '*'}

csiro_arccss: &csiro_arccss
    - {dataset: *, institute: CSIRO-ARCCSS, project: CMIP6, exp: amip, ensemble: *,
       <<: *timerange_amip, <<: *areacella_suppl, ...}
csiro: &csiro
    - {dataset: *, institute: CSIRO, project: CMIP6, exp: amip, ensemble: *,
       <<: *timerange_amip, <<: *areacella_suppl, ...}
bcc: &bcc
    - {dataset: *, institute: BCC, project: CMIP6, exp: amip, ensemble: *,
       <<: *timerange_amip, <<: *areacella_suppl, ...}
...


diagnostics:
    diagnostic_write_csv_and_run_lme:
        description: write the data collected by ESMValTool into a csv, then run the R LME analysis on it
        variables:
            global_annual_tas:
                short_name: tas
                mip: Amon
                preprocessor: global_annual_mean
                additional_datasets: # Only add the relevant entries for this variable here
                    - {<<: *csiro_arccss}
                    - {<<: *csiro}
                    - {<<: *bcc}
            global_annual_ts:
                short_name: ts
                mip: Amon
                preprocessor: global_annual_mean
                additional_datasets:
                    - {<<: *csiro_arccss}
                    - {<<: *bcc}
        scripts:
...

Note: the positioning of some parameters like supplementary_variables or timerange can be adapted to best fit what you find more readable and understandable, either directly inside the dataset definition or for each variable for example.

[bouweandela]

I would recommend writing a small script or Jupyter notebook to populate the datasets and additional_datasets sections of your recipe, that will be much less cumbersome. See here for an example that you can start out from: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/notebooks/composing-recipes.html

To find out which datasets you want to ignore, you can add a bit of code like this:

dataset_template = esmvalcore.dataset.Dataset(
    short_name="tas",
    mip="Amon",
    project="CMIP6",
    exp="historical",
    dataset="*",
    institute="*",
    ensemble="*",
    grid="*",
)
all_datasets = list(dataset_template.from_files())
usable_datasets = []
for dataset in results:
try:
    dataset.load()
except esmvalcore.cmor.check.CMORCheckError:
    pass
else:
    usable_datasets.append(dataset)
for dataset in usable_datasets:
    dataset.facets["diagnostic"] = "diagnostic_name"
print(yaml.safe_dump(esmvalcore.dataset.datasets_to_recipe(usable_datasets)))

[prosku]

Thanks @jlenh and @bouweandela, that's super helpful! I'm making progress with these answers and will report back once I have figured it out or have more concrete questions to ask.

[prosku]

Indeed, @bouweandela's answer above did the trick. I had to expand it a pit and am posting it here in case someone stumbles over a similar problem:

    all_datasets = list(dataset_template.from_files())
    usable_datasets = []
    for dataset in all_datasets:
        if (dataset.facets['dataset'] == 'ACCESS1-0' and dataset.facets['short_name'] in ['uas', 'sfcWind']) or (dataset.facets['dataset'] == 'HadGEM3-GC31-LM' and dataset.facets['short_name'] in ['vas', 'tauv']):
            print('This is one of the datasets that will fail later on and are thus excluded here')
            # exceptions for datasets that shouldn't be appended because they make the workflow fail later on (known issues I figure out over time), e.g. 
            # /work/bd0854/DATA/ESMValTool2/CMIP5_DKRZ/CSIRO-BOM/ACCESS1-0/amip/mon/atmos/Amon/r1i1p1/v20120727/sfcWind/sfcWind_Amon_ACCESS1-0_amip_r1i1p1_197901-200812.nc needs to be excluded because it has the wrong latitudes for its fx file (other variables are fine)
            continue
        try:
            d_tmp = dataset.load()
            print("Loaded")
        except esmvalcore.cmor.check.CMORCheckError:
            print("Passing CMORCheckError")
            pass
        except esmvalcore.esgf._download.DownloadError:
            print("Passing DownloadError")
            pass
        except esmvalcore.exceptions.InputFilesNotFound:
            print("Passing InputFilesNotFound exception")
            pass
        except ValueError:
            print("Passing ValueError")
            pass
        else:
            if len(d_tmp.coord('time').points) < 324:
                print("The dataset does not fill the full time period so not adding it to the usable list")
                #check whether each dataset covers the whole time range and don;t add it to the usable list if it doesn't
                continue
            else:
                usable_datasets.append(dataset)
                print("Appended")
    for dataset in usable_datasets:
        print("Renaming diagnostic_name")
        dataset.facets["diagnostic"] = "diagnostic_name"
    with open('recipes/searched_esgf_'+project+'_'+exp+'_'+var+'.yml', 'w') as file:
        yaml.safe_dump(esmvalcore.dataset.datasets_to_recipe(usable_datasets), file)

[prosku]

Regarding my original 2. point (the KeyError): I could indeed fix that in the ESMValCore. However, the change to /esmvalcore/cmor/_fixes/cmip6/ukesm1_0_ll.py is tiny:

             try:
                 if cube.attributes[parent_units] == bad_value:
                     cube.attributes[parent_units] = "days since 1850-01-01"
+            except KeyError:
+                pass
             except AttributeError:
                 pass
         return cubes

so I am wondering what the preferred way of contributing that is? @jlenh

[valeriupredoi]

you could compress that by changing to except KeyError or AttributeError: - best way to contribute is to open a Pull Request in ESMValCore please - and many thanks! 🍺

[prosku]

Thanks for the hint! I added a very small pull request: github.com/ESMValGroup/ESMValCore/pull/2782