jimmy-7664 / std-mae Goto Github PK

[IJCAI-24] Spatial-Temporal-Decoupled Masked Pre-training for Spatiotemporal Forecasting

Home Page: https://arxiv.org/abs/2312.00516

Python 99.86% Shell 0.14%

self-supervised-learning timeseries-forecasting

std-mae's Introduction

[IJCAI-24] Spatial-Temporal-Decoupled Masked Pre-training for Spatiotemporal Forecasting

Our work is already accepted by IJCAI2024 main track. The citation information will be updated when the official IJCAI24 proceeding is online.

Preprint Link (All six datasets [PEMS03, 04, 07, 08, PEMS-BAY, and METR-LA] are included.)

Google Scholar

Due to the modification of STD-MAE's title, you can simply search for "STD-MAE" in Google Scholar to get our article.

Citation

@article{gao2023spatio,
  title={Spatio-Temporal-Decoupled Masked Pre-training for Traffic Forecasting},
  author={Gao, Haotian and Jiang, Renhe and Dong, Zheng and Deng, Jinliang and Song, Xuan},
  journal={arXiv preprint arXiv:2312.00516},
  year={2023}
}

Performance on Spatiotemporal Forecasting Benchmarks

Please note you can get a much better performance on PEMS07 dataset using pre-training length of 2016. But it is a time-cosuming operation.

METR-LA	PEMS-BAY

💿 Dependencies

OS

Linux systems (e.g. Ubuntu and CentOS).

Python

The code is built based on Python 3.9, PyTorch 1.13.0, and EasyTorch. You can install PyTorch following the instruction in PyTorch.

Miniconda or Anaconda are recommended to create a virtual python environment.

We implement our code based on BasicTS.

Other Dependencies

pip install -r requirements.txt

Getting started

Download Data

You can download data from BasicTS and unzip it.

Preparing Data

Pre-process Data

You can pre-process all datasets by

cd /path/to/your/project
bash scripts/data_preparation/all.sh

Then the dataset directory will look like this:

datasets
   ├─PEMS03
   ├─PEMS04
   ├─PEMS07
   ├─PEMS08
   ├─raw_data
   |    ├─PEMS03
   |    ├─PEMS04
   |    ├─PEMS07
   |    ├─PEMS08
   ├─README.md

Pre-training on S-MAE and T-MAE

cd /path/yourproject

Then run the folloing command to run in Linux screen.

screen -d -m python stdmae/run.py --cfg='stdmae/TMAE_PEMS03.py' --gpus='0' 

screen -d -m python stdmae/run.py --cfg='stdmae/TMAE_PEMS04.py' --gpus='0'

screen -d -m python stdmae/run.py --cfg='stdmae/TMAE_PEMS07.py' --gpus='0' 

screen -d -m python stdmae/run.py --cfg='stdmae/TMAE_PEMS08.py' --gpus='0'

screen -d -m python stdmae/run.py --cfg='stdmae/SMAE_PEMS03.py' --gpus='0' 

screen -d -m python stdmae/run.py --cfg='stdmae/SMAE_PEMS04.py' --gpus='0'

screen -d -m python stdmae/run.py --cfg='stdmae/SMAE_PEMS07.py' --gpus='0' 

screen -d -m python stdmae/run.py --cfg='stdmae/SMAE_PEMS08.py' --gpus='0'

Downstream Predictor

After pre-training , copy your pre-trained best checkpoint to mask_save/. For example:

cp checkpoints/TMAE_200/064b0e96c042028c0ec44856f9511e4c/TMAE_best_val_MAE.pt mask_save/TMAE_PEMS04_864.pt
cp checkpoints/SMAE_200/50cd1e77146b15f9071b638c04568779/SMAE_best_val_MAE.pt mask_save/SMAE_PEMS04_864.pt

Then run the predictor as :

screen -d -m python stdmae/run.py --cfg='stdmae/STDMAE_PEMS04.py' --gpus='0' 

screen -d -m python stdmae/run.py --cfg='stdmae/STDMAE_PEMS03.py' --gpus='0' 

screen -d -m python stdmae/run.py --cfg='stdmae/STDMAE_PEMS08.py' --gpus='0'

screen -d -m python stdmae/run.py --cfg='stdmae/STDMAE_PEMS07.py' --gpus='0'

To find the best result in logs, you can search best_ in the log files.

std-mae's People

Contributors

Stargazers

Watchers

Forkers

zhuolinli-shu duyanhkt kurhula jeongwhanchoi afogsheep ironier mahmutatia

std-mae's Issues

Question about Downstream

Hi, there!

First, I would like to appreciate you for your opening source code. I'm very interested to the Downstream Spatiotemporal Predictor in your essay. However, I haven't found any code about it. Could you show me where is it?

如何引用这篇文章呢

你能提供吗老哥除了arxiv的

'STDMAERunner' object has no attribute 'to_running_device'

AttributeError: 'STDMAERunner' object has no attribute 'to_running_device'

May I ask why the upper and lower displays are different?

Only when the project is running, it shows that there are 0 GPUs, and when testing, there is one GPU.

预训练权重测试指标以及学习率问题

作者您好，
我对配置文件stdmae/TMAE_PEMS03.py以及相应权重mask_save\SMAE_PEMS03_864.pt运行BasicTS中的runner.test_process，得到结果：[test_MAE: 99.9056, test_RMSE: 134.5602, test_MAPE: 1.3745]；对配置文件stdmae/TMAE_PEMS04.py以及相应权重mask_save\SMAE_PEMS04_864.pt运行BasicTS中的runner.test_process，得到结果：[test_MAE: 113.2828, test_RMSE: 145.2737, test_MAPE: 1.7690]。这些测试结果中MAE指标远高于正常值，我本地训练TMAE_PEMS03 40轮的测试结果为[test_MAE: 13.3202, test_RMSE: 22.9870, test_MAPE: 0.1361]，请问你提供的mask_save权重是否有误？
此外，BatchSize=64或128的情况下TMAE训练过程中的MAE指标不太理想，调整学习率后MAE指标还是远高于作者BatchSize=4情况下的设置，BatchSize=8的情况下也效果不好，但是BatchSize=4情况下训练速度过慢（BatchSize=128设置下大概是116秒一轮，BatchSize=4时约五分钟），请问应该如何设置学习率以适应较大的BatchSize？
谢谢

I got the following error after changing GWNET to STID.

I adjusted the model and parameters to what STID requires, and during the debugging process, the tensor dimensions remained consistent, but this error cannot be modified.

Table 2 Results

Hello, I would like to confirm if the final results in Table2 is the average of the 12 predicted steps?

Question about Pretraining

Thank you for such good work, I like it!
However, when I reproduced the codes, I found the pretraining process is too long (5 minutes for 1 epoch), that means you need one day for the pretraining, is that right?
I reproduced the pretraining process based on GPU 3090.

jimmy-7664 / std-mae Goto Github PK

std-mae's Introduction

[IJCAI-24] Spatial-Temporal-Decoupled Masked Pre-training for Spatiotemporal Forecasting

Preprint Link (All six datasets [PEMS03, 04, 07, 08, PEMS-BAY, and METR-LA] are included.)

Google Scholar

Citation

Performance on Spatiotemporal Forecasting Benchmarks

💿 Dependencies

OS

Python

Other Dependencies

Getting started

Download Data

Preparing Data

Pre-training on S-MAE and T-MAE

Downstream Predictor

std-mae's People

Contributors

Stargazers

Watchers

Forkers

std-mae's Issues

Recommend Projects

Recommend Topics

Recommend Org