Skip to content

Commit

Permalink
Merge branch 'recipe_test_workflow_prototype' into 3246-rtw-container
Browse files Browse the repository at this point in the history
  • Loading branch information
alistairsellar authored Oct 30, 2023
2 parents 6e15941 + 036d1df commit 50553a8
Show file tree
Hide file tree
Showing 14 changed files with 380 additions and 175 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ jobs:
conda activate esmvaltool
mkdir -p ~/climate_data
esmvaltool config get_config_user
echo "offline: false" >> ~/.esmvaltool/config-user.yml
echo "search_esgf: when_missing" >> ~/.esmvaltool/config-user.yml
cat ~/.esmvaltool/config-user.yml
for recipe in esmvaltool/recipes/testing/recipe_*.yml; do
esmvaltool run "$recipe"
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/79bf6932c2e844eea15d0fb1ed7e415c)](https://www.codacy.com/gh/ESMValGroup/ESMValTool?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValTool&utm_campaign=Badge_Coverage)
[![Codacy Badge](https://app.codacy.com/project/badge/Grade/79bf6932c2e844eea15d0fb1ed7e415c)](https://www.codacy.com/gh/ESMValGroup/ESMValTool?utm_source=github.com&utm_medium=referral&utm_content=ESMValGroup/ESMValTool&utm_campaign=Badge_Grade)
[![Docker Build Status](https://img.shields.io/docker/cloud/build/esmvalgroup/esmvaltool.svg)](https://hub.docker.com/r/esmvalgroup/esmvaltool/)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/esmvaltool/badges/version.svg)](https://anaconda.org/conda-forge/esmvaltool)
[![Anaconda-Server Badge](https://img.shields.io/badge/Anaconda.org-2.9.0-blue.svg)](https://anaconda.org/conda-forge/esmvaltool)
![stand with Ukraine](https://badgen.net/badge/stand%20with/UKRAINE/?color=0057B8&labelColor=FFD700)

![esmvaltoollogo](https://raw.githubusercontent.com/ESMValGroup/ESMValTool/main/doc/sphinx/source/figures/ESMValTool-logo-2.png)
Expand Down
238 changes: 120 additions & 118 deletions conda-linux-64.lock

Large diffs are not rendered by default.

11 changes: 10 additions & 1 deletion doc/sphinx/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,16 @@
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
# html_theme_options = {}
#
# Avoid the following warning issued by pydata_sphinx_theme:
#
# "WARNING: The default value for `navigation_with_keys` will change to `False`
# in the next release. If you wish to preserve the old behavior for your site,
# set `navigation_with_keys=True` in the `html_theme_options` dict in your
# `conf.py` file.Be aware that `navigation_with_keys = True` has negative
# accessibility implications:
# https://github.com/pydata/pydata-sphinx-theme/issues/1492"
html_theme_options = {"navigation_with_keys": False}

# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions doc/sphinx/source/recipes/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ IPCC
recipe_ipccwg1ar6ch3
recipe_ipccwg1ar5ch9
recipe_collins13ipcc
recipe_examples

Land
^^^^
Expand Down
29 changes: 21 additions & 8 deletions doc/sphinx/source/recipes/recipe_examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,37 +8,43 @@ Overview

These are example recipes calling example diagnostic scripts.

The recipe examples/recipe_python.yml produces time series plots of global mean
The recipe ``examples/recipe_python.yml`` produces time series plots of global mean
temperature and for the temperature in Amsterdam.
It also produces a map of global temperature in January 2020.

The recipe examples/recipe_extract_shape.yml produces a map of the mean
The recipe ``examples/recipe_easy_ipcc.yml`` reproduces part of figure 9.3a from
`IPCC AR6 - Climate Change 2021: The Physical Science Basis <https://www.ipcc.ch/report/sixth-assessment-report-working-group-i/>`__.
It demonstrates how ESMValTool can be used to conveniently analyze
many models on their native grid and is described in detail in the blog post
`Analysis-ready climate data with ESMValCore <https://blog.esciencecenter.nl/easy-ipcc-powered-by-esmvalcore-19a0b6366ea7>`__.

The recipe ``examples/recipe_extract_shape.yml`` produces a map of the mean
temperature in the Elbe catchment over the years 2000 to 2002.
Some example shapefiles for use with this recipe are available
`here <https://github.com/ESMValGroup/ESMValTool/tree/main/esmvaltool/diag_scripts/shapeselect/testdata>`__,
make sure to download all files with the same name but different extensions.

The recipe examples/recipe_julia.yml produces a map plot with the mean temperature
The recipe ``examples/recipe_julia.yml`` produces a map plot with the mean temperature
over the year 1997 plus a number that is configurable from the recipe.

The recipe examples/recipe_decadal.yml showcases how the ``timerange`` tag
The recipe ``examples/recipe_decadal.yml`` showcases how the ``timerange`` tag
can be used to load datasets belonging to the DCPP activity. Produces timeseries
plots comparing the global mean temperature of a DCPP dataset with an observational
dataset.

Available recipes and diagnostics
---------------------------------

Recipes are stored in esmvaltool/recipes/

Recipes are stored in `esmvaltool/recipes/ <https://github.com/ESMValGroup/ESMValTool/tree/main/esmvaltool/recipes>`__:
* examples/recipe_python.yml
* examples/recipe_easy_ipcc.yml
* examples/recipe_extract_shape.yml
* examples/recipe_julia.yml
* examples/recipe_decadal.yml

Diagnostics are stored in esmvaltool/diag_scripts/

Diagnostics are stored in `esmvaltool/diag_scripts/ <https://github.com/ESMValGroup/ESMValTool/tree/main/esmvaltool/diag_scripts>`__:
* examples/diagnostic.py: visualize results and store provenance information
* examples/make_plot.py: Create a timeseries plot with likely ranges
* examples/diagnostic.jl: visualize results and store provenance information
* examples/decadal_example.py: visualize results and store provenance information

Expand All @@ -63,6 +69,7 @@ Variables
---------

* tas (atmos, monthly, longitude, latitude, time)
* tos (ocean, monthly, longitude, latitude, time)

Example plots
-------------
Expand All @@ -79,6 +86,12 @@ Example plots

Amsterdam air temperature (multimodel mean of CMIP5 CanESM2 and CMIP6 BCC-ESM1).

.. _easy_ipcc:
.. figure:: /recipes/figures/examples/IPCC_AR6_figure_9.3a_1850-2100.png
:align: center

Mean sea surface temperature anomaly (part of figure 9.3a from IPCC AR6).

.. _elbe:
.. figure:: /recipes/figures/examples/elbe.png
:align: center
Expand Down
76 changes: 76 additions & 0 deletions esmvaltool/diag_scripts/examples/make_plot.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
"""Python example diagnostic."""
import logging
from pathlib import Path

import iris
import matplotlib.pyplot as plt

from esmvaltool.diag_scripts.shared import run_diagnostic, save_figure

logger = logging.getLogger(Path(__file__).stem)


def main(cfg):
"""Plot part of figure_9.3a from IPCC AR6."""
colors = {
'historical-ssp126': '#2a3652',
'historical-ssp585': '#78333a',
}
fill_colors = {
'historical-ssp126': '#d2d5dc',
'historical-ssp585': '#ddced2',
}
labels = {
'historical-ssp126': 'Historical and SSP1-2.6',
'historical-ssp585': 'Historical and SSP5-8.5',
}

# Group input data by experiment
groups = {}
for filename, attributes in cfg['input_data'].items():
exp = attributes['exp']
if exp not in groups:
groups[exp] = {}
groups[exp][attributes['dataset']] = filename

# Loop over experiments to populate plot
for exp, group in groups.items():
mean = iris.load_cube(group['MultiModelMean'])
iris.quickplot.plot(
mean,
color=colors.get(exp),
label=labels.get(exp, exp),
)

p17 = iris.load_cube(group['MultiModelP17'])
p83 = iris.load_cube(group['MultiModelP83'])
time_coord = mean.coord('time')
time_axis = time_coord.units.num2date(time_coord.core_points())
plt.fill_between(
time_axis,
p17.core_data(),
p83.core_data(),
color=fill_colors.get(exp),
label='Likely (17% - 83%) ranges',
)

plt.title('Sea surface temperature anomaly')
plt.legend(loc='upper left')

filename = 'IPCC_AR6_figure_9.3a_1850-2100'
provenance_record = {
'caption': "Part of figure 9.3a from IPCC AR6.",
'authors': [
'kalverla_peter',
'andela_bouwe',
],
'references': ['fox-kemper21ipcc'],
'ancestors': list(cfg['input_data'].keys()),
}
save_figure(filename, provenance_record, cfg, dpi=300)


if __name__ == '__main__':

with run_diagnostic() as config:
main(config)
126 changes: 126 additions & 0 deletions esmvaltool/recipes/examples/recipe_easy_ipcc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
documentation:
title: Easy IPCC
description: Reproduce part of IPCC AR6 figure 9.3a.
references:
- fox-kemper21ipcc
authors:
- kalverla_peter
- andela_bouwe
maintainer:
- andela_bouwe

preprocessors:
easy_ipcc:
custom_order: true
anomalies:
period: month
reference:
start_year: 1950
start_month: 1
start_day: 1
end_year: 1979
end_month: 12
end_day: 31
area_statistics:
operator: mean
annual_statistics:
operator: mean
convert_units:
units: 'degrees_C'
ensemble_statistics:
statistics:
- mean
multi_model_statistics:
statistics:
- mean
- p17
- p83
span: full
keep_input_datasets: false
ignore_scalar_coords: true

diagnostics:
AR6_Figure_9.3:
variables:
tos_ssp585:
short_name: tos
exp: ['historical', 'ssp585']
project: CMIP6
mip: Omon
preprocessor: easy_ipcc
timerange: '1850/2100'
tos_ssp126:
short_name: tos
exp: ['historical', 'ssp126']
project: CMIP6
mip: Omon
timerange: '1850/2100'
preprocessor: easy_ipcc
scripts:
Figure_9.3a:
script: examples/make_plot.py

datasets:
- {dataset: ACCESS-CM2, ensemble: 'r(1:5)i1p1f1', grid: gn}
- {dataset: ACCESS-ESM1-5, ensemble: 'r(1:40)i1p1f1', grid: gn}
- {dataset: AWI-CM-1-1-MR, ensemble: r1i1p1f1, grid: gn}
- {dataset: BCC-CSM2-MR, ensemble: r1i1p1f1, grid: gn}
# - {dataset: CAMS-CSM1-0, ensemble: 'r(1:2)i1p1f1', grid: gn} # available data does not fully cover timerange
- {dataset: CAS-ESM2-0, ensemble: r1i1p1f1, grid: gn}
- {dataset: CAS-ESM2-0, ensemble: r3i1p1f1, grid: gn}
- {dataset: CESM2, ensemble: r4i1p1f1, grid: gn}
- {dataset: CESM2, ensemble: 'r(10:11)i1p1f1', grid: gn}
- {dataset: CESM2-WACCM, ensemble: r1i1p1f1, grid: gn}
- {dataset: CIESM, ensemble: r1i1p1f1, grid: gn}
- {dataset: CMCC-CM2-SR5, ensemble: r1i1p1f1, grid: gn}
- {dataset: CMCC-ESM2, ensemble: r1i1p1f1, grid: gn}
- {dataset: CNRM-CM6-1, ensemble: 'r(1:6)i1p1f2', grid: gn}
- {dataset: CNRM-CM6-1-HR, ensemble: r1i1p1f2, grid: gn}
- {dataset: CNRM-ESM2-1, ensemble: 'r(1:5)i1p1f2', grid: gn}
- {dataset: CanESM5, ensemble: 'r(1:25)i1p(1:2)f1', grid: gn}
- {dataset: CanESM5-1, ensemble: 'r1i1p(1:2)f1', grid: gn, institute: CCCma}
- {dataset: CanESM5-CanOE, ensemble: 'r(1:3)i1p2f1', grid: gn}
- {dataset: EC-Earth3, ensemble: r1i1p1f1, grid: gn}
- {dataset: EC-Earth3, ensemble: r4i1p1f1, grid: gn}
- {dataset: EC-Earth3, ensemble: r6i1p1f1, grid: gn}
# - {dataset: EC-Earth3, ensemble: r9i1p1f1, grid: gn} # download failure of ssp585
- {dataset: EC-Earth3, ensemble: r11i1p1f1, grid: gn}
- {dataset: EC-Earth3, ensemble: r15i1p1f1, grid: gn}
# - {dataset: EC-Earth3, ensemble: 'r(101:150)i1p1f1', grid: gn} # available data does not fully cover timerange
- {dataset: EC-Earth3-Veg, ensemble: 'r(1:4)i1p1f1', grid: gn}
- {dataset: EC-Earth3-Veg, ensemble: r6i1p1f1, grid: gn}
# - {dataset: EC-Earth3-Veg-LR, ensemble: 'r(1:3)i1p1f1', grid: gn} # mismatch between i and j coordinate names between historical and ssp experiment
- {dataset: FGOALS-f3-L, ensemble: 'r(1:3)i1p1f1', grid: gn}
- {dataset: FGOALS-g3, ensemble: 'r(1:4)i1p1f1', grid: gn}
- {dataset: FIO-ESM-2-0, ensemble: 'r(1:3)i1p1f1', grid: gn}
- {dataset: GFDL-ESM4, ensemble: r1i1p1f1, grid: gn}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:4)i1p5f1', grid: gn, supplementary_variables: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p5f1}, {short_name: areacello, skip: true}]}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:5)i1p1f2', grid: gn, supplementary_variables: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p1f1}, {short_name: areacello, skip: true}]}
- {dataset: GISS-E2-1-G, ensemble: 'r(1:5)i1p3f1', grid: gn, supplementary_variables: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p3f1}, {short_name: areacello, skip: true}]}
- {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p1f2', grid: gn, supplementary_variables: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p1f1}, {short_name: areacello, skip: true}]}
- {dataset: GISS-E2-1-H, ensemble: 'r(1:5)i1p3f1', grid: gn, supplementary_variables: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p3f1}, {short_name: areacello, skip: true}]}
- {dataset: GISS-E2-2-G, ensemble: 'r(1:5)i1p3f1', grid: gn, supplementary_variables: [{short_name: areacella, mip: fx, exp: piControl, ensemble: r1i1p1f1}, {short_name: areacello, skip: true}]}
- {dataset: HadGEM3-GC31-LL, ensemble: r1i1p1f3, grid: gn}
- {dataset: HadGEM3-GC31-MM, ensemble: r1i1p1f3, grid: gn}
# - {dataset: IITM-ESM, ensemble: r1i1p1f1, grid: gn} # available data does not fully cover timerange
- {dataset: INM-CM4-8, ensemble: r1i1p1f1, grid: gr1}
- {dataset: INM-CM5-0, ensemble: r1i1p1f1, grid: gr1}
- {dataset: IPSL-CM6A-LR, ensemble: 'r(1:4)i1p1f1', grid: gn}
- {dataset: IPSL-CM6A-LR, ensemble: r6i1p1f1, grid: gn}
- {dataset: IPSL-CM6A-LR, ensemble: r14i1p1f1, grid: gn}
# - {dataset: KACE-1-0-G, ensemble: 'r(1:3)i1p1f1', grid: gr} # unstructured grid but no cell area information available
# - {dataset: KIOST-ESM, ensemble: r1i1p1f1, grid: gr1} # historical and ssp126 experiment are on different grids
- {dataset: MCM-UA-1-0, ensemble: r1i1p1f2, grid: gn}
- {dataset: MIROC-ES2H, ensemble: r1i1p4f2, grid: gn}
- {dataset: MIROC-ES2L, ensemble: 'r(1:10)i1p1f2', grid: gn}
- {dataset: MIROC6, ensemble: 'r(1:50)i1p1f1', grid: gn}
- {dataset: MPI-ESM1-2-HR, ensemble: 'r1i1p1f1', grid: gn}
# - {dataset: MPI-ESM1-2-HR, ensemble: 'r(1:2)i1p1f1', grid: gn} # second ensemble member causes warnings about large graphs in `concatenate` preprocessor step
- {dataset: MPI-ESM1-2-LR, ensemble: 'r(1:30)i1p1f1', grid: gn}
- {dataset: MRI-ESM2-0, ensemble: 'r(1:5)i1p1f1', grid: gn}
# - {dataset: NESM3, ensemble: 'r(1:2)i1p1f1', grid: gn} # cannot be used due to https://github.com/ESMValGroup/ESMValCore/issues/2101
# - {dataset: NorESM2-LM, ensemble: r1i1p1f1, grid: gn} # duplicated areacello file with wrong name
- {dataset: NorESM2-MM, ensemble: r1i1p1f1, grid: gn}
- {dataset: TaiESM1, ensemble: r1i1p1f1, grid: gn}
- {dataset: UKESM1-0-LL, ensemble: 'r(1:4)i1p1f2', grid: gn}
- {dataset: UKESM1-0-LL, ensemble: r8i1p1f2, grid: gn}
2 changes: 1 addition & 1 deletion esmvaltool/recipes/recipe_heatwaves_coldwaves.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ datasets:
# - {dataset: IPSL-CM5A-MR, type: exp, project: CMIP5, exp: rcp85,
# ensemble: r1i1p1, start_year: 2020, end_year: 2040}
- {dataset: bcc-csm1-1, type: exp, project: CMIP5, exp: historical,
ensemble: r1i1p1, start_year: 1971, end_year: 2000}
ensemble: r1i1p1, start_year: 1971, end_year: 2000, version: v1}
- {dataset: bcc-csm1-1, type: exp, project: CMIP5, exp: rcp85,
ensemble: r1i1p1, start_year: 2060, end_year: 2080}

Expand Down
11 changes: 11 additions & 0 deletions esmvaltool/references/fox-kemper21ipcc.bibtex
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
@inbook{fox-kemper21ipcc,
author={B. Fox-Kemper and H.T. Hewitt and C. Xiao and G. Aðalgeirsdóttir and S.S. Drijfhout and T.L. Edwards and N.R. Golledge and M. Hemer and R.E. Kopp and G. Krinner and A. Mix and D. Notz and S. Nowicki and I.S. Nurhati and L. Ruiz and J.-B. Sallée and A.B.A. Slangen and Y. Yu},
editor={V. Masson-Delmotte and P. Zhai and A. Pirani and S.L. Connors and C. Pean and S. Berger and N. Caud and Y. Chen and L. Goldfarb and M.I. Gomis and M. Huang and K. Leitzell and E. Lonnoy and J.B.R. Matthews and T.K. Maycock and T. Waterfield and O. Yelekci and R. Yu and B. Zhou},
title={Ocean, Cryosphere and Sea Level Change},
booktitle={Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change},
year={2021},
publisher={Cambridge University Press},
address={Cambridge, UK and New York, NY, USA},
pages = {1211--1362},
doi={10.1017/9781009157896.011}
}
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,14 @@ The |RTW| performs the following steps:

``configure``
:Description:
Creates and modifies the |ESMValTool| user configuration file
Creates the |ESMValTool| user configuration file
:Runs on:
Localhost
:Executes:
The ``configure.py`` script from the |Rose| app
:Details:
``configure`` should run at the start of each cycle after ``clone_latest_esmval``
has completed; in the case where the next cycle doesn't happen for a week,
the clone could potentially be out of date by the time of the task trigger.
``configure`` should run at the start of each cycle after
``install_env_file`` has completed.

``process``
:Description:
Expand Down
Loading

0 comments on commit 50553a8

Please sign in to comment.