CMIP6 extesnions for UKESM2

[antony-john-payne]

I am using ESMvaltool to look at CMIP6 extensions to 2300. I get an error for UKESM2 which the ESGF indicates ran to 2300. For example

{dataset: UKESM1-0-LL, project: CMIP6, exp: ssp585, ensemble: r4i1p1f2, grid: gn, timerange: 2250/2299}

ESGF suggests the data exist

but I get the error message suggesting data not found

2026-01-13 15:02:57,552 UTC [1794879] ERROR Could not create all tasks
2026-01-13 15:02:57,552 UTC [1794879] ERROR Missing data for preprocessor plot_multiple_timeseries/tas:

• Missing data for Dataset: tas, Amon, CMIP6, UKESM1-0-LL, ScenarioMIP, ssp585, r4i1p1f2, gn, supplementaries: areacella, fx, CMIP, piControl, r1i1p1f2

Is this something to do with the supplementaries (not certain where r1i1p1f2 came from)?

Code works for the other nine ESMs that ran SSP585 to ~2300 - CanESM5, MRI-ESM2-0, IPSL-CM6A-LR, ACCESS-CM2, ACCESS-ESM1-5, CESM2-WACCM, EC-Earth3-Veg, GISS-E2-1-H, NorESM2-MM.

Any ideas?

Thanks tony

[valeriupredoi]

hi @antony-john-payne thanks for dropping by - I had a chat with Robin about this - we'll have a look at the recipe and how it runs (or doesn't, for the matter) tomorrow afternoon, then I'll get back to you here after I figure out what the problem is, as it stands now I can't do much without the recipe 🍺

[antony-john-payne]

Hi @valeriupredoi thnaks for this. Please note that I am running ESMvaltool on a wrokstation rather than JASMIN. I think Robin ran the recipe successfully on JASMIN.

[valeriupredoi]

indeed, that's why I want to grab hold of the recipe, and run it locally myself, see if there are any issues with remote data finding and downloading 🍺

[antony-john-payne]

this is the version of Robin's recipe that I have been working on. the first 6 ESMs work then fails for NOrESM and others. have check that data exist on ESGF checking end time of expt, ensemble, grid etc.

recipe_ISMIP7.yml

[valeriupredoi]

excellent, cheers, Tony! Can I please confirm you are running the bogstandard esmvaltool==2.13 from eg conda-forge? If you could run conda list esmval for me pls, and post the output (that should include the ESMValCore version too) 🍺

[antony-john-payne]

packages in environment at /home/antpayne/miniforge3/envs/esmvaltool:

Name Version Build Channel

esmvalcore 2.13.0 pyhcf101f3_0 conda-forge
esmvaltool 2.13.0 pyh22db52d_2 conda-forge
esmvaltool-ncl 2.13.0 hb13dca2_2 conda-forge
esmvaltool-python 2.13.0 pyhcf101f3_2 conda-forge
esmvaltool-r 2.13.0 hb13dca2_2 conda-forge
esmvaltool-sample-data 0.0.4 pyhd8ed1ab_0 conda-forge

[valeriupredoi]

perfect, thanks! I'll have a test run then 🍺

[antony-john-payne]

I sue a set of shapefiles from GMT that you may need to comment out

[valeriupredoi]

@antony-john-payne I ran a test and indeed the UKESM data is unavailable for download for me as well, though - some data is available, see a more limited 2050/2100 timerange actually works, and the data gets downloaded:

ESGFFile:CMIP6/ScenarioMIP/MOHC/UKESM1-0-LL/ssp585/r4i1p1f2/Amon/tas/gn/v20190507/tas_Amon_UKESM1-0-LL_ssp585_r4i1p1f2_gn_201501-204912.nc on hosts ['esgf3.dkrz.de']
ESGFFile:CMIP6/ScenarioMIP/MOHC/UKESM1-0-LL/ssp585/r4i1p1f2/Amon/tas/gn/v20190507/tas_Amon_UKESM1-0-LL_ssp585_r4i1p1f2_gn_205001-210012.nc on hosts ['esgf3.dkrz.de']
2026-01-16 15:20:25,412 UTC [2797502] DEBUG   esmvalcore.esgf._search:378 Selected files:
ESGFFile:CMIP6/ScenarioMIP/MOHC/UKESM1-0-LL/ssp585/r4i1p1f2/Amon/tas/gn/v20190507/tas_Amon_UKESM1-0-LL_ssp585_r4i1p1f2_gn_205001-210012.nc on hosts ['esgf3.dkrz.de']

this is most probably due to data being available to download via the front-end API (like you posted above, and it returned correct search hits), but the same data is not available anymore via the esgf-pyclient downloader - @bouweandela what do you reckon?

As I see it, you got two options:

• either try downloading the data manually via the portal (front end), and pop it in climate_data together with all the other datasets for the recipe, or
• ping the ESGF folks via the mailinglist and ask them what's up with this particular dataset - they may have just unmounted it for some reason

[antony-john-payne]

Hi V. Thanks. The problem seems to come and go. Previously I was able to download 9/10 models (UKESM2 being the exception) using ESMvaltool but when I tried this again recently I could only get 6. Good to know that I wasn't doing anything silly. I guess I could also copy from JASMIN directly. do you have the ESGF mailinglist address? tony

[valeriupredoi]

BTW CCing @robinsmith-ncas here too 🍺

[valeriupredoi]

Hi V. Thanks. The problem seems to come and go. Previously I was able to download 9/10 models (UKESM2 being the exception) using ESMvaltool but when I tried this again recently I could only get 6. Good to know that I wasn't doing anything silly. I guess I could also copy from JASMIN directly. do you have the ESGF mailinglist address? tony

if you set search_esgf: when_missing in your config-user, then it should attempt to download only when the data is missing from your local climate_data dir, so once downloaded, it can stay there and be found; but yeah, ESGF is going through overhaul for ESGF2 now, so it's a bit of a game of hide and seek with data; indeed, the mailinglist is ESGF-USER@listserv.llnl.gov

[bouweandela]

You can try using a different index node to search for the data as described here: https://docs.esmvaltool.org/projects/ESMValCore/en/v2.13.0/quickstart/configure.html#configuration-file.

[valeriupredoi]

You can try using a different index node to search for the data as described here: https://docs.esmvaltool.org/projects/ESMValCore/en/v2.13.0/quickstart/configure.html#configuration-file.

@bouweandela indeed! This config file did the thing:

search_connection:
  urls:
    - 'https://esgf-node.ornl.gov/esgf-1-5-bridge'
    - 'https://esgf.ceda.ac.uk/esg-search'
    # - 'https://esgf-data.dkrz.de/esg-search'
    - 'https://esgf-node.ipsl.upmc.fr/esg-search'
    - 'https://esg-dn1.nsc.liu.se/esg-search'
    - 'https://esgf.nci.org.au/esg-search'
    - 'https://esgf.nccs.nasa.gov/esg-search'
    - 'https://esgdata.gfdl.noaa.gov/esg-search'
  distrib: true
  timeout: 120  # seconds
  cache: '~/.esmvaltool/cache/pyesgf-search-results'
  expire_after: 86400  # cache expires after 1 day

Note that we have a wee bug in the docs page (those ..)
The node that's found all the data is esgf-node.ornl.gov

[antony-john-payne]

I am afraid that did not work for me. Same errors. I tried commenting out all of the nodes except esgf-node.ornl.gov and got the following error

FileNotFoundError: Failed to search ESGF, unable to connect:

• 422 Client Error: Unprocessable Content for url: https://esgf-node.ornl.gov/esgf-1-5-bridge/?format=application%2Fsolr%2Bjson&limit=500&distrib=true&offset=10000&type=File&project=CMIP6&source_id=CanESM5&grid_label=gn&table_id=fx&variable=areacella

Perhaps I can't acess esgf-node.ornl.gov for some reason?