projectpythia / cmip6-cookbook Goto Github PK

View Code? Open in Web Editor NEW

11.0 2.0 9.0 32.92 MB

Examples of analysis of Google Cloud CMIP6 data using Pangeo tools

Home Page: https://projectpythia.org/cmip6-cookbook/

License: Apache License 2.0

Jupyter Notebook 98.59% CSS 0.18% HTML 1.23%

cmip6-cookbook's Introduction

CMIP6 Cookbook

This Project Pythia Cookbook covers examples of analysis of Google Cloud CMIP6 data using Pangeo tools.

Motivation

From the CMIP6 website:

The simulation data produced by models under previous phases of CMIP have been used in thousands of research papers ... and the multi-model results provide some perspective on errors and uncertainty in model simulations. This information has proved invaluable in preparing high profile reports assessing our understanding of climate and climate change (e.g., the IPCC Assessment Reports).

With such a large amount of model output produced, moving the data around is inefficient. In this collection of notebooks, you will learn how to access cloud-optimized CMIP6 datasets, in addition to a few examples of using that data to analyze some aspects of climate change.

Authors

Ryan Abernathey, Henri Drake, Robert Ford, Max Grover

Contributors

Structure

Foundations

This section includes three variations of accessing CMIP6 data from cloud storage.

Example workflows

There are currently four examples of using this data to

Estimate equilibrium climate sensitivity (ECS)
Plot global mean surface temperature under two different Shared Socioeconomic Pathways
Plot changes in precipitation intensity under the SSP585 scenario
Calculate changes in ocean heat uptake after regridding with xESMF

Running the Notebooks

You can either run the notebook using Binder or on your local machine.

Running on Binder

The simplest way to interact with a Jupyter Notebook is through Binder, which enables the execution of a Jupyter Book in the cloud. The details of how this works are not important for now. All you need to know is how to launch a Pythia Cookbooks chapter via Binder. Simply navigate your mouse to the top right corner of the book chapter you are viewing and click on the rocket ship icon, (see figure below), and be sure to select “launch Binder”. After a moment you should be presented with a notebook that you can interact with. I.e. you’ll be able to execute and even change the example programs. You’ll see that the code cells have no output at first, until you execute them by pressing {kbd}Shift+{kbd}Enter. Complete details on how to interact with a live Jupyter notebook are described in Getting Started with Jupyter.

Running on Your Own Machine

If you are interested in running this material locally on your computer, you will need to follow this workflow:

Clone the https://github.com/ProjectPythia/cmip6-cookbook repository:
```
 git clone https://github.com/ProjectPythia/cmip6-cookbook.git
```
Move into the cmip6-cookbook directory
```
cd cmip6-cookbook
```
Create and activate your conda environment from the environment.yml file
```
conda env create -f environment.yml
conda activate cmip6-cookbook-dev
```
Move into the notebooks directory and start up Jupyterlab
```
cd notebooks/
jupyter lab
```

At this point, you can interact with the notebooks! Make sure to check out the "Getting Started with Jupyter" content from the Pythia Foundations material if you are new to Jupyter or need a refresher.

cmip6-cookbook's People

Contributors

Stargazers

Watchers

Forkers

ktyle brian-rose jukent r-ford esgf2-us mgrover1 r-ford2 hdrake jsta

cmip6-cookbook's Issues

Need Henri Drake's ORCID

We now have a CITATION.cff file for author attribution. It would be great to add an ORCID (and maybe other metadata) for @hdrake.

Henri, if you see this, please feel free to open a PR to add your details.

Typos

This is a running list of typos to fix at some point. Feel free to edit this comment with more.

intake-esm.ipynb:

in Overview: "packages is" → "package is"

Code in `precip-freq.ipynb` not executing during build

ERROR: Execution Failed: /home/runner/work/cmip6-cookbook/cmip6-cookbook/notebooks/example-workflows/precip-freq.ipynb

This might be because it takes too long; there are four plots and one took about 15 minutes. The notebook runs fine on the binder hub.

Examples with secrets not executing

What happened:
Since the new ARM/ESGF example uses github secrets, we are running into issues passing those credentials to the execution workflow on the binderhub

What you expected to happen:
We should be able to execute notebooks with secrets through the CI system, especially since binderbot explicitly allows this.

Anything else we need to know?:
I opened a PR to resolve this over in ProjectPythia/cookbook-actions#50

Progress bars in published notebooks

When there is a progress bar in a notebook, each update gets written as a new line in the published version of the notebook. I think this makes notebooks more difficult to read. For example, this notebook is probably ~90% progress bar.

Is there any way to have the progress bar only one line (or just removed)?

Add text to "Summary" and "What's next?" sections in each notebook

Revert back from esgf examples

I accidentally pushed to the wrong repo... working on reverting changes to the config/esgf api example file.

Add Julius's Amazing Xmip + CMIP6 AMS Notebook

@jbusecke 's presentation was awesome, and his notebook would be a great addition to the CMIP6 cookbook

Would you be interested in contributing this to the Pythia CMIP6 cookbook @jbusecke ?

Link to the notebook

Add image and authors to `README.md`

As in the template

Logo is missing

On this page https://cookbooks.projectpythia.org/cmip6-cookbook/README.html
the logo seems to be missing

Reduce `col_wrap` in `ecs-cmip6.ipynb`

The ECS notebook currently uses 7 models, and with col_wrap=5, the plots are smaller than they need to be.

Add the ability to run on the ESGF JupyterHub (US-side)

I think it would be good to enable the Jupyterhub launch to nimbus, the ESGF (US-side of the project) resource.

Currently, we have this in a fork of the repository, but I think it would be better to be within the main version.

Change the 4xCO2 wording

ecs-cmip6.ipynb currently uses the 2xCO2 runs instead of 4xCO2 in order to run quicker, so the commentary should reflect this.

Errors in ENSO ESGF notebook

The ENSO ESGF notebook is currently failing to build. There is a line with invalid syntax. When I removed that line in my fork, another issue came up, which can be seen here, but I can't figure out what this new error means.

Build is failing at imports

See https://github.com/ProjectPythia/cmip6-cookbook/actions/runs/3624041278/jobs/6110575382#step:11:819

Link checker failing

The nightly build is currently failing because of the link checker. There are 403 (Forbidden) errors for links to a few papers, which are all AGU publications.

Odd units in temperature plot

The temperature timeseries at the end of this section
doesn't seem to have sensible temperature units.

Error when executing code in binder

I was trying out the CMIP6 cookbook but after binder launch the first cell execution will show a error like this

---------------------------------------------------------------------------
SystemError                               Traceback (most recent call last)
Cell In[1], line 1
----> 1 import xarray as xr
      2 xr.set_options(display_style='html')
      3 import intake

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/__init__.py:1
----> 1 from . import testing, tutorial
      2 from .backends.api import (
      3     load_dataarray,
      4     load_dataset,
   (...)
      8     save_mfdataset,
      9 )
     10 from .backends.rasterio_ import open_rasterio

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/testing.py:9
      6 import numpy as np
      7 import pandas as pd
----> 9 from xarray.core import duck_array_ops, formatting, utils
     10 from xarray.core.dataarray import DataArray
     11 from xarray.core.dataset import Dataset

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/duck_array_ops.py:28
     26 from . import dask_array_ops, dtypes, npcompat, nputils
     27 from .nputils import nanfirst, nanlast
---> 28 from .pycompat import cupy_array_type, is_duck_dask_array
     29 from .utils import is_duck_array
     31 try:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/pycompat.py:48
     44         self.type = duck_array_type
     45         self.available = duck_array_module is not None
---> 48 dsk = DuckArrayModule("dask")
     49 dask_version = dsk.version
     50 dask_array_type = dsk.type

File /srv/conda/envs/notebook/lib/python3.9/site-packages/xarray/core/pycompat.py:27, in DuckArrayModule.__init__(self, mod)
     24 duck_array_version = Version(duck_array_module.__version__)
     26 if mod == "dask":
---> 27     duck_array_type = (import_module("dask.array").Array,)
     28 elif mod == "pint":
     29     duck_array_type = (duck_array_module.Quantity,)

File /srv/conda/envs/notebook/lib/python3.9/importlib/__init__.py:127, in import_module(name, package)
    125             break
    126         level += 1
--> 127 return _bootstrap._gcd_import(name[level:], package, level)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/array/__init__.py:2
      1 try:
----> 2     from dask.array import backends, fft, lib, linalg, ma, overlap, random
      3     from dask.array.blockwise import atop, blockwise
      4     from dask.array.chunk_types import register_chunk_type

File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/array/backends.py:18
     16 from dask.array.numpy_compat import divide as np_divide
     17 from dask.array.numpy_compat import ma_divide
---> 18 from dask.array.percentile import _percentile
     19 from dask.backends import CreationDispatch, DaskBackendEntrypoint
     21 concatenate_lookup.register((object, np.ndarray), np.concatenate)

File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/array/percentile.py:9
      6 import numpy as np
      7 from tlz import merge
----> 9 from dask.array.core import Array
     10 from dask.array.numpy_compat import _numpy_122
     11 from dask.array.numpy_compat import percentile as np_percentile

File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/array/core.py:36
     34 from dask.array import chunk
     35 from dask.array.chunk import getitem
---> 36 from dask.array.chunk_types import is_valid_array_chunk, is_valid_chunk_type
     38 # Keep einsum_lookup and tensordot_lookup here for backwards compatibility
     39 from dask.array.dispatch import (  # noqa: F401
     40     concatenate_lookup,
     41     einsum_lookup,
     42     tensordot_lookup,
     43 )

File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/array/chunk_types.py:122
    119     pass
    121 try:
--> 122     import sparse
    124     register_chunk_type(sparse.SparseArray)
    125 except ImportError:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/sparse/__init__.py:1
----> 1 from ._coo import COO, as_coo
      2 from ._compressed import GCXS
      3 from ._dok import DOK

File /srv/conda/envs/notebook/lib/python3.9/site-packages/sparse/_coo/__init__.py:1
----> 1 from .core import COO, as_coo
      2 from .common import (
      3     concatenate,
      4     clip,
   (...)
     22     diagonalize,
     23 )
     25 __all__ = [
     26     "COO",
     27     "as_coo",
   (...)
     47     "diagonalize",
     48 ]

File /srv/conda/envs/notebook/lib/python3.9/site-packages/sparse/_coo/core.py:9
      6 import warnings
      8 import numpy as np
----> 9 import numba
     10 import scipy.sparse
     11 from numpy.lib.mixins import NDArrayOperatorsMixin

File /srv/conda/envs/notebook/lib/python3.9/site-packages/numba/__init__.py:43
     39 from numba.core.decorators import (cfunc, generated_jit, jit, njit, stencil,
     40                                    jit_module)
     42 # Re-export vectorize decorators and the thread layer querying function
---> 43 from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
     44                             get_num_threads, set_num_threads)
     46 # Re-export Numpy helpers
     47 from numba.np.numpy_support import carray, farray, from_dtype

File /srv/conda/envs/notebook/lib/python3.9/site-packages/numba/np/ufunc/__init__.py:3
      1 # -*- coding: utf-8 -*-
----> 3 from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
      4 from numba.np.ufunc._internal import PyUFunc_None, PyUFunc_Zero, PyUFunc_One
      5 from numba.np.ufunc import _internal, array_exprs

File /srv/conda/envs/notebook/lib/python3.9/site-packages/numba/np/ufunc/decorators.py:3
      1 import inspect
----> 3 from numba.np.ufunc import _internal
      4 from numba.np.ufunc.parallel import ParallelUFuncBuilder, ParallelGUFuncBuilder
      6 from numba.core.registry import TargetRegistry

SystemError: initialization of _internal failed without raising an exception

The error will be gone if import xarray as xr is removed. Does anyone have the same problem? Thank you!

Python / Dask issues

@r-ford, good news and bad:

Good: I was able to get the ecs-cmip6 notebook to launch, with no import errors, after I specified python<3.10 in the environment.yml file
Bad: The very next cell failed:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [2], in <cell line: 1>()
----> 1 gateway = Gateway()
      2 cluster = gateway.new_cluster()
      3 cluster.adapt(minimum=1, maximum=20)

File ~/miniforge3/envs/cmip6-cookbook-dev/lib/python3.9/site-packages/dask_gateway/client.py:282, in Gateway.__init__(self, address, proxy_address, public_address, auth, asynchronous, loop)
    280     address = format_template(dask.config.get("gateway.address"))
    281 if address is None:
--> 282     raise ValueError(
    283         "No dask-gateway address provided or found in configuration"
    284     )
    285 address = address.rstrip("/")
    287 if public_address is None:

ValueError: No dask-gateway address provided or found in configuration

projectpythia / cmip6-cookbook Goto Github PK

cmip6-cookbook's Introduction

CMIP6 Cookbook

Motivation

Authors

Contributors

Structure

Foundations

Example workflows

Running the Notebooks

Running on Binder

Running on Your Own Machine

cmip6-cookbook's People

Contributors

Stargazers

Watchers

Forkers

cmip6-cookbook's Issues

Recommend Projects

Recommend Topics

Recommend Org