Git Product home page Git Product logo

pangeo-openeo-bids-2023's Introduction

Unlocking the Power of Space Data with Pangeo & OpenEO

All Contributors

This repository contains the documentation and jupyter notebooks used for delivering tutorial to BiDS'23.


The content of this repository (folder tutorial) is rendered as an online document using Jupyter Book. You can access it here.

Timeline of the workshops

The programmes for each workshop are given below for your information. Each workshop is held separately.

Introduction to Pangeo

Time Activity
09:00 πŸ‘‹ Welcome (5 minutes)
09:05 Introduction and Motivation (15 minutes)
09:20 Β Overview of the Pangeo ecosystem (20 minutes)
Β 09:40 Β Understanding Xarray to avoid common pitfalls (30 minutes)
Β 10:10 Β Interactive Visualization with Hvplot (15 minutes)
10:30 Break (30 minutes)

Introduction to OpenEO

Time Activity
11:00 πŸ‘‹ Introduction and motivation (5 minutes)
11:05 Getting started with OpenEO (10 minutes)
11:15 Accessing and processing data with OpenEO (30 minutes)
11:45 Integrate custom code into your workflow using User Defined Functions (30 minutes)
12:15 Q&A session - feedbacks (15 minutes)
12:30 🍽️ Lunch

Unlocking the Power of Space Data with Pangeo & OpenEO

Please note that this workshop assumes some prior knowledge of Pangeo and OpenEO. If you are not familiar with any of these technologies, we suggest to check the content of the two other workshops (taught in the morning).

Time Activity
14:00 πŸ‘‹ Introduction and motivation (5 minutes)
14:05 Data discoverability and searchability (25 minutes)
An overview of STAC and the different available sources
14:30 Data and pre-processing general knowledge (60 minutes)
Introduction to chunking with netCDF, ZARR and Kerchunk
Parallelization with Dask
15:30 β˜•οΈ Break
16:00 Different data exploitation approaches (60 minutes)
How to exploit data with OpenEO: snow coverage example
How to exploit data on Pangeo: pure Xarray version
17:00 How to go beyond (45 minutes)
Custom algorithms: UDF (OpenEO) and ufunc (Xarray)
Scaling with OpenEO (how it works underneath)
17:40 Wrap-up and feedback survey (15 minutes)

These timelines are purely approximative and given for indication purpose only. We will adjust depending on the audience. There will be additional breaks (5 minutes) regularly and time for questions during the workshops.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Alejandro Β©
Alejandro Β©

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Alexander Jacob
Alexander Jacob

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Amin Razbani
Amin Razbani

πŸ€” πŸ““
Anne Fouilloux
Anne Fouilloux

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Daniel Thiex
Daniel Thiex

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Guillaume Eynard-Bontemps
Guillaume Eynard-Bontemps

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Jeroen Dries
Jeroen Dries

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Justus Magin
Justus Magin

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Matthias Mohr
Matthias Mohr

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Michele Claus
Michele Claus

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Miguel Caballer Fernandez
Miguel Caballer Fernandez

πŸš‡ πŸ“¦
Ola
Ola

πŸ€”
Pratichhya
Pratichhya

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Sebastian Luna-Valero
Sebastian Luna-Valero

πŸš‡ πŸ“¦
Stefaan Lippens
Stefaan Lippens

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
Tina Odaka
Tina Odaka

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
ValentinaHutter
ValentinaHutter

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…
ZdenΔ›k Ε ustr
ZdenΔ›k Ε ustr

πŸš‡ πŸ“¦
bschumac
bschumac

πŸš‡ πŸ“¦ πŸ“‹
j34ni
j34ni

πŸš‡ πŸ“¦
pl.marasco
pl.marasco

πŸ€” 🎨 πŸ’» πŸ–‹ πŸ“– βœ…

This project follows the all-contributors specification. Contributions of any kind welcome!

pangeo-openeo-bids-2023's People

Contributors

acocac avatar allcontributors[bot] avatar annefou avatar bschumac avatar clausmichele avatar dthiex avatar felixcremer avatar jdries avatar keewis avatar pl-marasco avatar tinaok avatar valentinahutter avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pangeo-openeo-bids-2023's Issues

create link to do git pull

We would like to have a link to do git pull automatically as we had for foss4g tutorial session.
On foss4g, we had a link as follows;

https://pangeo-foss4g.vm.fedcloud.eu/jupyterhub/hub/user-redirect/git-pull?repo=https%3A//github.com/pangeo-data/foss4g-2022&urlpath=lab/tree/foss4g-2022/tutorial/pangeo101/&branch=main

So I imagine that we need something like this.

https://pangeo-eosc.vm.fedcloud.eu/jupyterhub/hub/user-redirect/git-pull?repo=https%3A//github.com/pangeo-data/pangeo-openeo-BiDS-2023&urlpath=lab/tree/pangeo-openeo-BiDS-2023/tutorial/part1/&branch=main

But it is not working. I think for foss4g @guillaumeeb made it happen, does anyone have any idia how to make it work?

file: part3/chunking_introduction

  • Update obsolete commands
  • Put visibilities of 'Zarr' in table of contents and/or title
  • Fix warning on hdf.SingleHdf5ToZarr
  • update example for exporting zarr,
  • add intake
  • put intake catalogue on s3? update file path name update from foss4g to ?

Various notebooks issues

@pl-marasco @acocac

In a new environment with Python 3.10 and latest xarray 2023.9.0 (probably something changed recently) I'm getting this kind of errors in various notebooks:

import xarray as xr

catalogue="https://object-store.cloud.muni.cz/swift/v1/foss4g-catalogue/c_gls_NDVI-LTS_1999-2019.json"
LTS = xr.open_mfdataset(
    "reference://", engine="zarr",
    backend_kwargs={
        "storage_options": {
            "fo":catalogue
                    },
        "consolidated": False
    }
)
LTS

Error:

File ~/anaconda3/envs/stac_310/lib/python3.10/site-packages/zarr/storage.py:1382, in FSStore.__init__(self, url, normalize_keys, key_separator, mode, exceptions, dimension_separator, fs, check, create, missing_exceptions, **storage_options)
   1380 else:
   1381     if storage_options:
-> 1382         raise ValueError("Cannot specify both fs and storage_options")
   1383     self.fs = fs
   1384     self.path = self.fs._strip_protocol(url)

ValueError: Cannot specify both fs and storage_options

GAUL broken link

          In the `xarray_hands_on` notebook, I'm getting this error:

DriverError: '/vsizip/vsicurl/https://mars.jrc.ec.europa.eu/asap/files/gaul1_asap.zip' does not exist in the file system, and is not recognized as a supported dataset name.

Not sure if it's due to my environment.

Originally posted by @clausmichele in #6 (comment)

file: part3/scaling_dask

  • update dask from 2022-> now, verify that runs in the updated versions.
  • change path of ../figures to ./figures, and move figures to part3/figures/xx, update figures.

TOC Overview

This an overview of the proposed TOC according to #14. We can open separate issues by converting each of the checkboxes to issues.

  • caption: About
    chapters:

    • file: about/timeline
  • caption: Before the workshop
    chapters:

    • #26
    • file: before/free_trial
  • caption: Part 1 - Pangeo
    chapters:

    • #17
    • file: part1/overview
    • file: part1/xarray_pitfalls
    • file: part1/visualization
  • caption: Part 2 - OpenEO

    • file: part2/agenda_and_links
    • file: part2/advanced_workflows
    • file: part2/stac_metadata
  • caption: Part 3
    chapters:

    • file: part3/data_discovery
    • file: part3/data_exploitability_openEO
    • file: part3/data_exploitability_pangeo
    • #19
    • #23
    • file: part3/scaling_openEO
    • file: part3/advanced_udf
    • #31
    • #28
  • caption: Beyond the workshop
    chapters:

    • file: afterword/resources
    • #92
    • file: afterword/envds-book

In the `wildfires` notebook there's an issue with the used STAC Catalog from Element84, since the v0 is no more updated.

          In the `wildfires` notebook there's an issue with the used SATC Catalog from Element84, since the v0 is no more updated.
# prefire data
time_range = "2021-08-10/2021-08-16"

query1 = catalog.search(
    collections=["sentinel-s2-l2a-cogs"], datetime=time_range, limit=100,
    bbox=bbox, query={"eo:cloud_cover": {"lt": 0.5}},
)

items = list(query1.get_items())
print(f"Found: {len(items):d} datasets")

items_pre = min(items, key=lambda item: item.properties["eo:cloud_cover"])

prefire_ds = stac_load(
    [items_pre],
    bands=("B08", "B12"),
    crs=crs,
    resolution=10 * zoom,
    chunks={},  # <-- use Dask
    groupby="datetime",
    bbox=bbox,
)

returns:

DoesNotConformTo: Server does not conform to ITEM_SEARCH, There is not fallback option available for search.

We need to switch to this one:

catalog = Client.open("https://earth-search.aws.element84.com/v1")

with this collection

query1 = catalog.search(
    collections=["sentinel-2-l2a"], datetime=time_range, limit=100,
    bbox=bbox, query={"eo:cloud_cover": {"lt": 0.5}},
)

band names are also different:

bands=("nir", "swir22"),

Originally posted by @clausmichele in #6 (comment)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.