marccoru / crop-type-mapping Goto Github PK

View Code? Open in Web Editor NEW

151.0 8.0 43.0 35.1 MB

Source code to Rußwurm & Körner 2019. Self-Attention for Raw Optical Satellite Time Series Classification

License: MIT License

Dockerfile 0.01% Shell 0.08% Jupyter Notebook 98.54% Python 1.37%

crop-type-mapping's Introduction

Self-Attention for Raw Optical Satellite Time Series Classification

Feature extraction through self-attention on Raw Sentinel 2 Time Series

Source Code of Rußwurm & Körner (2019), Self-Attention for Raw Optical Satellite Time Series Classification

@article{russwurm2020,
title = "Self-attention for raw optical Satellite Time Series Classification",
journal = "ISPRS Journal of Photogrammetry and Remote Sensing",
volume = "169",
pages = "421 - 435",
year = "2020",
issn = "0924-2716",
doi = "https://doi.org/10.1016/j.isprsjprs.2020.06.006",
url = "http://www.sciencedirect.com/science/article/pii/S0924271620301647",
author = "Marc Rußwurm and Marco Körner",
}

Architectures

Four deep learning models for time series classification are implemented

Recurrent Neural Net (LSTM) src/models/rnn.py
Transformer src/models/transformerEncoder.py
TempCNN (Pelleter et al., 2019) src/models/tempcnn.py
Multi-scale ResNet src/models/msresnet.py

Getting started

Python Dependencies

Anaconda environment

conda create -n crop-type-mapping python=3.7.3 pip
conda activate crop-type-mapping
pip install -r requirements.txt

Download Dataset, Tuning Results and Models

download raw Sentinel 2 crop type label dataset to data/BavarianCrops via

bash download.sh dataset

and the pretrained models for hyperparameter tuning and evaluation via

bash download.sh models

or both with bash download.sh all

Experiments are composed of a set of parameters that define model and datasets. Experiments are defined in the if-else cases in src/experiments.py and generally follow the naming convention isprs_<model>_<dataset>_<meta>.

Training

start visdom server by running visdom in conda environment and open http://localhost:8097/ in the browser.

train from scratch (23 classes) with hyperparameters defined in ../models/tune/23classes/transformer_tum.csv. Working directory is src

python train.py --experiment isprs_tum_transformer \
    --store /tmp/ \
    --classmapping ../data/BavarianCrops/classmapping23.csv \
    --hyperparameterfolder ../models/tune/23classes

to continue training with 23 classes point --store to an existing model

python train.py --experiment isprs_tum_transformer \
    --store ../models/train/23classes/0 \
    --classmapping ../data/BavarianCrops/classmapping23.csv \
    --hyperparameterfolder ../models/tune/23classes

experiments on raw dataset: isprs_tum_transformer, isprs_tum_msresnet, isprs_tum_tempcnn, isprs_tum_rnn

classmappings: mapping tables to select 12 or 23 classes to classify

hyperparameter folder: folder with results of ray-tune results implemented in tune.py. hyperparameters summarized in csv files for model and dataset, as defined in src/hyperparameters.py

Hyperparameter Tuning

Ray-Tune allows hyperparameter tuning of multiple models in parallel. Execute in src

python tune.py --experiment transformer_tum \
    -g 0.5 -c 2 -b 64 --local_dir ../models/tune --seed 0 \
    --classmapping $PWD/../data/BavarianCrops/classmapping23.csv \
    --dataroot $PWD/../data

to train two models per GPU and store results in ../models/tune. Valid experiments are [transformer|rnn|msresnet|tempcnn]_[tum|gaf] which are defined in src/hyperparameter.py

External Code

Self-Attention implementation by Yu-Hsiang Huang

https://github.com/jadore801120/attention-is-all-you-need-pytorch

Multi-scale ResNet implementation by Fei Wang

https://github.com/geekfeiw/Multi-Scale-1D-ResNet

Original Tensorflow TempCNN implementation by Charlotte Pelletier

https://github.com/charlotte-pel/igarss2019-dl4sits

crop-type-mapping's People

Contributors

Stargazers

Watchers

crop-type-mapping's Issues

Datasets

Hi, I don't know how to download the GAFdatasets in the download.sh. It will cause the problem in 01_datasets.

~\Anaconda3\envs\py38\lib\site-packages\tables\file.py in init(self, filename, mode, title, root_uep, filters, **kwargs)
776
777 # Now, it is time to initialize the File extension
--> 778 self._g_new(filename, mode, **params)
779
780 # Check filters and set PyTables format version for new files.

tables\hdf5extension.pyx in tables.hdf5extension.File._g_new()

~\Anaconda3\envs\py38\lib\site-packages\tables\utils.py in check_file_access(filename, mode)
152 # The file should be readable.
153 if not os.access(filename, os.F_OK):
--> 154 raise IOError("%s does not exist" % (filename,))
155 if not os.path.isfile(filename):
156 raise IOError("%s is not a regular file" % (filename,))

OSError: /data/GAFdataset\test_train_holl.h5 does not exist

R2 Visualization of Classification results plus ground truth

In the experiments, the authors should visualize some classification results by using different deep learning models for more intuitive comparison. Besides, please also provide some visualization results of the ground truth dataset.

R1: Shallow Baseline Models CCDC BFAST Timesat

The shallow baseline method is based on Random Forest classifier (Page 6, Section 4.5).
However it is not clear how it is applied, especially what is the input data fed to the random
forest classifier : is it a temporal feature vector, or is a classification performed for every image
then some kind of majority voting along the time series ? Furthermore, there are many shallow
learning, time series analysis methods that have been applied to crop mapping. It would be
more fair to evaluate a proposed deep learning time series method vs a state-of-the art,
classical satellite image time series method, such as CCDC, BFAST or Timesat.

R1: More Baseline Models Duplo

The proposed model includes
a self-attention mechanism on top of Transformer model, that should be compared to another
attention mechanism be it a softer one. For example the one implemented in DuPLO [44] (code
can be obtained by contacting the authors). DuPLO is also a combination of RNN and CNN
and a model that was applied to crop classification with Sentinel-2 time series, which makes it
a very relevant baseline method for these three reasons. Please include additional experiments
with DuPLO.

R1 Add UMap Dimensionality reduction to T-SNE experiment

Page 14 Section 8.4 and Figure 7 : the t-SNE visualization is a good idea. However since tSNE, other non-linear dimensionality reduction methods that outperform t-SNE have been
proposed and implemented such as UMAP: https://github.com/lmcinnes/umap UMAP often
performs better at preserving aspects of global structure of the data than t-SNE. This means
that it can often provide a better "big picture" view of your data as well as preserving local
neighbour relations. You should give it a try. UMAP has been used also in remote sensing
publications, such as Mohammadimanesh et al., 2019 ISPRS Journal of Photogrammetry and
Remote Sensing, https://www.sciencedirect.com/science/article/abs/pii/S092427161930084X

how to get notebookdata

When I click the notebookdata URL 'https://syncandshare.lrz.de/dl/fiM6b3e7eeyFAGWmAHEeoeBB/notebookdata.zip', it shows HTTP ERROR 404. I download these data on Windows platform, so I can only obtain these data manually.