Git Product home page Git Product logo

py_madaclim's Introduction

py-madaclim: a simple Python API to interact with the Madaclim database

GitHub top language

Table of Contents
  1. Package Description
  2. Installation
  3. OS-specific steps
  4. General workflow
  5. Getting started (quick example)
  6. References
  7. Contact us

py-madaclim Package Description

Example of a MadaclimCollection plot with layer 79

py-madaclim is a Python 3+ package that allows to interact with the Madaclim db, an open-source climate and environmental database for Madagascar.

Fetch and explore the raster-based data with metadata information support, create new datasets from existent spreadsheets/csv/dataframes from any Coordinate Reference System (CRS) and explore/manipulate your data with visualization and transformation tools.

If you prefer Read-The-Docs style documentation go here.

API documentation is also available.

Installation

Environment setup and requirements

py-madaclim is working with Python 3.10 and 3.11. For now, we have provided two ways to setup a working environment for both versions:

  • Using pip and venv for Python=3.10
  • Using Conda for Python=3.11

The requirements for the conda setup can be found in conda_requirements.txt and for the pip setup in venv_requirements.txt. OS-specific installation steps are listed below:

Debian/Linux/macOS systems

Steps for pip installation (Recommended)

  1. Clone the repo and create a new venv
  2. git clone https://github.com/tahiri-lab/py_madaclim.git
    cd py_madaclim
    python -m venv ~/.pyenv/py_mada_env    #python=3.10
    source ~/.pyenv/py_mada_env/bin/activate
  3. Activate the environment and install the requirements
  4. pip install -r venv_requirements.txt    # reqs before py_madaclim
    pip install .    # to install py-madaclim

Steps for conda installation (Slower option)

  1. First follow these instructions to install conda on your machine
  2. Clone the repo and configure the conda-forge channel
  3. git clone https://github.com/tahiri-lab/py_madaclim.git
    cd py_madaclim
    
    # Configure correct channel priority in ~/.condarc
    conda config --add channels conda-forge && conda config --append channels plotly
    conda config --show channels
    # channels:
    #   - conda-forge
    #   - defaults
    #   - plotly
  4. Create the environment with dependencies (slow step, be patient!)
  5. conda create -n py_mada_env --file conda_requirements.txt
  6. Activate the environment and install py-madaclim
  7. conda activate py_mada_env
    pip install .    # using pip inside conda env

Windows

  1. Clone the repo and create a new venv
  2. git clone https://github.com/tahiri-lab/py_madaclim.git
    cd py_madaclim
    python -m venv ~/.pyenv/py_mada_env    #python=3.10
    ./.pyenv/py_mada_env/Scripts/activate
  3. Activate the environment and install the requirements
  4. pip install -r venv_requirements.txt    # reqs before py_madaclim
    pip install .    # to install py-madaclim

Getting Started: Quick Example

For a full walkthrough, follow along this notebook

Madaclim db metadata with the info module

Basic metada and download rasters from Madaclim server

# Get available methods and properties for MadaclimLayers
>>> from py_madaclim.info import MadaclimLayers
>>> mada_info = MadaclimLayers()
>>> print(mada_info)
MadaclimLayers(
	all_layers = DataFrame(79 rows x 6 columns)
	categorical_layers = DataFrame(Layers 75, 76, 77, 78 with a total of 79 categories
	public methods -> download_data, fetch_specific_layers, get_categorical_combinations
			 get_layers_labels, select_geoclim_type_layers
)

# To access all layers as a dataframe
>>> mada_info.all_layers
geoclim_type  layer_number layer_name                       layer_description  is_categorical    units
0         clim             1      tmin1   Monthly minimum temperature - January           False  °C x 10
...

# Built-in method to download the Madaclim raster files
>>> mada_info.download_data(save_dir=cwd)

Get detailed labels for each raster layers

>>> env_labels = mada_info.get_layers_labels(
    layers_subset="env", 
    as_descriptive_labels=True
)
>>> print(env_labels[0])
'env_71_alt_Altitude (meters)'

Explore the rasters and create datasets with the raster_manipulation module

MadaclimRasters basic properties and visualization methods

>>> from py_madaclim.raster_manipulation import MadaclimRasters

>>> mada_rasters = MadaclimRasters("madaclim_current.tif", "madaclim_enviro.tif")
>>> print(mada_rasters)
MadaclimRasters(
	clim_raster = madaclim_current.tif,
	clim_crs = epsg:32738,
	clim_nodata_val = -32768.0
	env_raster = madaclim_enviro.tif,
	env_crs = epsg:32738,
	env_nodata_val = -32768.0
)

# Basic visualization for a continuous data layer
>>> mada_rasters.plot_layer(
...     layer=env_labels[0], 
...     imshow_cmap="terrain", 
...     histplot_binwidth=100, histplot_stat="count", 
... )
dasdas

Create sample points with MadaclimPoint and MadaclimCollection

>>> from py_madaclim.raster_manipulation import MadaclimPoint

# Single point
>>> specimen_1 = MadaclimPoint(specimen_id="abbayesii", longitude=46.8624, latitude=-24.7541)

# Multipoints
>>> coll = MadaclimCollection.populate_from_csv("collection_example.csv")
>>> print(coll[0])
MadaclimPoint(
	specimen_id = ABA,
	source_crs = 4326,
	longitude = 46.8624,
	latitude = -24.7541,
	mada_geom_point = POINT (688328.2403248843 7260998.022932809),
	sampled_layers = None (Not sampled yet),
	nodata_layers = None (Not sampled yet),
	is_categorical_encoded = False,
	Species = C.abbayesii,
	Botanical_series = Millotii,
	Genome_size_2C_pg = 1.25,
	gdf.shape = (1, 11)
)

Sample the rasters, visualize and encode the data for ML-related tasks

# Sample the collection reflects the changes to the geodataframe
>>> coll.sample_from_rasters(
...     clim_raster=mada_rasters.clim_raster,
...     env_raster=mada_rasters.env_raster,
...     layers_to_sample="all",   # Or any single/list of layers labels
...     layer_info=True
... )
>>> coll.gdf["specimen_id", env_labels[-1]]
specimen_id Percentage of forest cover in 1 km by 1 km grid cells (%)
ABA 100
AMB 65
ANK1 20
BISS 89
COS 0
VOHE 88
# Visualize on the raster map
>>> coll.plot_on_layer(env_labels[-1], imshow_cmap="coolwarm")
Collection plot example

Binary encoding for downstream ML applications

# One Hot encoding updates the object dynamically
>>> coll.binary_encode_categorical()
>>> print(coll.is_categorical_encoded)
True
>>> coll_categ_layers = set(["_".join(label.split("_")[:4]) for label in coll.encoded_categ_labels])
>>> print(f"Splitted {len(coll_categ_layers)} layers into {len(coll.encoded_categ_labels)} unique categories")
Splitted 4 layers into 83 unique categories

# Updated geodataframe attribute
>>> env_76_encoded = coll.encoded_categ_labels[12:30]
>>> coll.gdf[["specimen_id"] + env_76_encoded]
specimen_id env_76_soi_Soil types_Alluvio-colluvial_Deposited_Soils env_76_soi_Soil types_Andosols env_76_soi_Soil types_Bare_Rocks env_76_soi_Soil types_Fluvio-marine_Deposited_Soils_-_Mangroves env_76_soi_Soil types_Highly_Rejuvenated,_Penevoluted_Ferralitic_Soils env_76_soi_Soil types_Humic_Ferralitic_Soils env_76_soi_Soil types_Humic_Rejuvenated_Ferralitic_Soils env_76_soi_Soil types_Hydromorphic_Soils env_76_soi_Soil types_Indurated-Concretion_Ferralitic_Soils env_76_soi_Soil types_Podzolic_Soils_and_Podzols env_76_soi_Soil types_Poorly_Evolved_Erosion_Soils,_Lithosols env_76_soi_Soil types_Raw_Lithic_Mineral_Soils env_76_soi_Soil types_Red_Ferruginous_Soils env_76_soi_Soil types_Red_Fersiallitic_Soils env_76_soi_Soil types_Rejuvenated_Ferralitic_Soils_with_Degrading_Structure env_76_soi_Soil types_Rejuvenated_Ferralitic_Soils_with_Little_Degrading_Structure env_76_soi_Soil types_Salty_Deposited_Soils env_76_soi_Soil types_Skeletal_Shallow_Eroded_Ferruginous_Soils
ABA 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
AMB 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0
ANK1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0
BISS 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1
COS 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
VOHE 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

GBIF API utilities for pre-data fetching in the utils module

Request an occurence search and download the data

>>> from py_madaclim.utils import gbif_api

# Get taxonKey of interest
>>> coffea_key = gbif_api.get_taxon_key_by_species_match("coffea")
EXACT match type found with 95% confidence!
canonical name of match: Coffea
GBIF_taxon_key: 2895315

# Search occurrences
recent_years = (2010, 2023)
>>> coffea_search_results_2010_present = gbif_api.search_occ_mdg_valid_coordinates(
...     taxon_key=coffea_key,
...     year_range=recent_years
... )
Fetching all 613 occurences in year range 2010-2023...
Extracting occurences 0 to 300...
Extracting occurences 300 to 600...
Extracting occurences 600 to 613...
Total records retrieved: 613

# ...Or create a download for a given search
>>> from dotenv import load_dotenv
>>> import os
>>> load_dotenv(".env")
True
>>> download_id = gbif_api.request_occ_download_mdg_valid_coordinates(
...     taxon_key=coffea_key,
...     email=your_email@gmail.com,
...     year_range=recent_years    # Defaults to None which is all possible years
... )

# Download, extract and read as df
>>> coffea_gbif_df = gbif_api.download_extract_read_occ(
    download_id=download_id,
    target_dir="gbif_example"
)
Response OK from https://api.gbif.org/v1/occurrence/download for the given 'download_id'
Progress for download_0008397-230810091245214.zip : 100.0% completed of 0.21 MB downloaded [ average speed of 0.41 MB/s ]
Extracting all 17 files to target location: .../download_0008397-230810091245214/
Read and saved core data into pandas df: occurrence.txt

Create a MadaclimCollection from the GBIF occurrences

# Keep relevant data
>>> df = coffea_gbif_df.loc[coffea_gbif_df["taxonRank"] == "SPECIES"]
>>> df = df.loc[:, ["verbatimScientificName", "decimalLongitude", "decimalLatitude", "year"]]
>>> df = df.reset_index().drop(columns="index")
>>> df["specimen_id"] = df.apply(lambda row: f"{row['verbatimScientificName']}_{row.name}", axis=1)
>>> df["specimen_id"] = df["specimen_id"].str.strip("Coffea ")
>>> # Format for MadaclimCollection constructor
>>> df.columns = ["genus_species", "longitude", "latitude", "year", "specimen_id"]
>>> df.head()
genus_species longitude latitude year specimen_id
Coffea perrieri 46.015693 -17.117573 2023 perrieri_0
Coffea pervilleana 45.920397 -17.077081 2023 pervilleana_1
Coffea pervilleana 45.923007 -17.078820 2023 pervilleana_2
Coffea boiviniana (Baill.) Drake 49.353747 -12.336711 2020 boiviniana (Baill.) Drake_3
Coffea humbertii J.-F.Leroy 44.690055 -22.888583 2018 humbertii J.-F.Leroy_4
# Create a collection from recent samples
recent_coffea = MadaclimCollection.populate_from_df(df.loc[df["year"] >= 2020])

References

Contact Us

For any questions, feedback or to get in touch with us : [email protected]

For our lab's other research projects, visit our website

py_madaclim's People

Contributors

carofo73 avatar simlal avatar tahirinadia avatar

Stargazers

 avatar  avatar

Watchers

 avatar

Forkers

carofo73

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.