Update data links once LLNL -> ORNL migration is complete

[znichollscr]

Once these have moved, we should update the various search links. If it's easy, also worth potentially adding download links somewhere so people can get the direct download link even if metagrid is down.

[digbyr]

Do you know what the current status of the migration is? I'm looking for the SLCF forcings, and while I can view them on both LLNL and ORNL nodes, neither allows me to download the files. Direct HTTP downloads result in a "no available server" error, and the auto-generated wget scripts either cannot find any files to download (ORNL) or fail with "error 503: service not available" (LLNL). None of the international ESGF nodes appear to have the 2025-04-18 CEDS files.

[durack1]

Do you know what the current status of the migration is? I'm looking for the SLCF forcings, and while I can view them on both LLNL and ORNL nodes, neither allows me to download the files. Direct HTTP downloads result in a "no available server" error, and the auto-generated wget scripts either cannot find any files to download (ORNL) or fail with "error 503: service not available" (LLNL). None of the international ESGF nodes appear to have the 2025-04-18 CEDS files.

Noting this is a dupe of JGCRI/CEDS#68. As noted there @digbyr just checking, when I click this link I get 52 results found. When I click the top result here this file downloads for me (very quickly actually). Is that not the case for you?

[durack1]

Once these have moved, we should update the various search links.

If this works as planned, there will be no requirement to update the search links, the index will be updated to reflect the new host, which means all indexes pointing to the US index (LLNL, ORNL) which have a valid (and functioning) link.

The remaining SOLR indexes (DKRZ, CEDA (which doesn't appear to have input4MIPs configured)) will no longer work, if they are not updated to point to the new host (which could be the NERSC globus endpoint here, e.g. NOx-em-anthro_input4MIPs_emissions_CMIP_CEDS-CMIP-2025-04-18_gr_198701-198712.nc or their own replicated data holdings, which DKRZ has most of the recently published data).

[durack1]

None of the international ESGF nodes appear to have the 2025-04-18 CEDS files.

@digbyr this statement doesn't appear correct, DKRZ does in fact have this data replicated and republished locally - see here. It also seems that CEDA has replicated and republished this data too - see here.

Let me know if you don't have exactly the same experience as me - all nodes I have tested appear to be working for me

[digbyr]

Thanks @durack1 - maybe there was a hiccup on the ESGF side this morning, or maybe it was an issue on our end. Downloads seem to be working properly now.

Regarding the DKRZ node: I've figured out the issue. If I search from the input4mips project I only find the CMIP6 versions. The All (except CMIP6) project does indeed contain the CMIP7 forcings I'm looking for, but they don't appear if you select institute = PNNL-JGCRI which is how I had filtered the results. The file metadata does include institution_id: PNNL-JGCRI but maybe something about "institute" vs "institution_id" isn't keying properly to the dropdown menus?

[durack1]

If I search from the input4mips project I only find the CMIP6 versions.

I am a little confused by the above. I presume you are using the metagrid interface to search? If yes, then you have numerous search facets. MIP era = CMIP7 are the final datasets ready for simulations to begin, everything else is either previous MIP era (CMIP6) data, or CMIP6Plus prototype/testing data.

To calibrate, see the URLs below:

MIP era/CMIP phase	input4MIPs ESGF URL
CMIP7 (CMIP7 final data)	https://aims2.llnl.gov/search?project=input4MIPs&activeFacets={"mip_era":"CMIP7"}
CMIP6Plus (CMIP7 prototype data)	https://aims2.llnl.gov/search?project=input4MIPs&activeFacets={"mip_era":"CMIP6Plus"}
CMIP6 (CMIP6 final data)	https://aims2.llnl.gov/search?project=input4MIPs&activeFacets={"mip_era":"CMIP6"}

[durack1]

or maybe it was an issue on our end

@digbyr it is not obvious to me who you are, or where you are based, so without wild speculation, I can't do much to diagnose and identify a possible cause of any outage

[digbyr]

If I search from the input4mips project I only find the CMIP6 versions.
I am a little confused by the above. I presume you are using the metagrid interface to search?

Yes, this is with the metagrid interface, searching using the dropdown menus. On my screen, I have to select a project from the dropdown menu before I can search by variable or any other filter, and "CMIP7" is not one of the available options. So I checked input4mips and All (not CMIP6) as the most likely candidates; the CMIP7 CEDS forcings are under the latter project, but apparently not keyed to the PNNL-JGRI institute filter.

I can't do much to diagnose and identify a possible cause of any outage

Yes, not a problem! It's working now. Thanks very much for looking into it and sorry for the false alarm.

[durack1]

Ruth Digby (@digbyr - found you https://profils-profiles.science.gc.ca/en/profile/dr-ruth-digby), ok so you're connecting from CCCma/Canada.

I am a little confused still, as we have no "institute" filter - "Institution ID", sure, see below

If it wasn't obvious, you can force a search through metagrid by customizing your URL with search entries, so if you wanted PNNL-JGCRI data for the input4MIPs project, and MIP Era = CMIP7 - use this

https://aims2.llnl.gov/search?project=input4MIPs&activeFacets={"mip_era":"CMIP7","institution_id":"PNNL-JGCRI"}

If you want to add the latest source_id = CEDS-CMIP-2025-04-18 add in ,"source_id":"CEDS-CMIP-2025-04-18". If you want to target a specific variable, sure, no problem, add ,"variable_id":"SO2_em_anthro", etc etc.

If you want to check another ESGF node, no problem, switch out the LLNL node "aims2.llnl.gov" with the ORNL (Tennessee) node "esgf-node.ornl.gov" or the DKRZ (German) node "esgf-metagrid.cloud.dkrz.de", or the IPSL (French) node "esgf-node.ipsl.upmc.fr/projects/esgf-ipsl/" (note old interface), etc etc

[digbyr]

Thanks for the tips on customizing a URL, that's really useful!

Just for completeness, there is an institute filter on the All (except CMIP6) project:

and an institution_id filter on the input4mips project:

On the inputs4mips project, setting institution_id=PNNL-JGCRI returns forcings from CMIP6, CMIP6plus, and CMIP7. On the All (except CMIP6) project, setting institute=PNNL-JGCRI returns forcings from CMIP6 only.

[durack1]

Just for completeness, there is an institute filter on the All (except CMIP6) project:

Ah ok, we're unpeeling this onion slowly. I would never approach and ESGF search without defining a project=input4MIPs (for this use case) first. There have been vocabulary changes over the phases and projects which make cross-project searches a bit messy. In the input4MIPs project, these changes have been relatively minor, so is an easier project to navigate (hopefully).

FYI, a targeted project search can be targeted with the custom URL: https://aims2.llnl.gov/search?project=input4MIPs, you can also toggle out for CMIP3, CMIP5, CMIP6 projects (model data), and other projects like obs4MIPs too. You can then sub-target as well, see the links above