Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
Time series forecasting is a critical demand for real applications. Enlighted by the classic time series analysis and stochastic process theory, we propose the Autoformer as a general series forecasting model [paper]. Autoformer goes beyond the Transformer family and achieves the series-wise connection for the first time.
In long-term forecasting, Autoformer achieves SOTA, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease.
🚩News (2023.08) Autoformer has been included in Hugging Face. See blog.
🚩News (2023.06) The extension version of Autoformer (Interpretable weather forecasting for worldwide stations with a unified deep model) has been published in Nature Machine Intelligence as the Cover Article.
🚩News (2023.02) Autoformer has been included in our [Time-Series-Library], which covers long- and short-term forecasting, imputation, anomaly detection, and classification.
🚩News (2022.02-2022.03) Autoformer has been deployed in 2022 Winter Olympics to provide weather forecasting for competition venues, including wind speed and temperature.
1. Deep decomposition architecture
We renovate the Transformer as a deep decomposition architecture, which can progressively decompose the trend and seasonal components during the forecasting process.
Figure 1. Overall architecture of Autoformer.
2. Series-wise Auto-Correlation mechanism
Inspired by the stochastic process theory, we design the Auto-Correlation mechanism, which can discover period-based dependencies and aggregate the information at the series level. This empowers the model with inherent log-linear complexity. This series-wise connection contrasts clearly from the previous self-attention family.
Figure 2. Auto-Correlation mechansim.
- Install Python 3.6, PyTorch 1.9.0.
- Download data. You can obtain all the six benchmarks from Google Drive. All the datasets are well pre-processed and can be used easily.
- Train the model. We provide the experiment scripts of all benchmarks under the folder
./scripts. You can reproduce the experiment results by:
bash ./scripts/ETT_script/Autoformer_ETTm1.sh
bash ./scripts/ECL_script/Autoformer.sh
bash ./scripts/Exchange_script/Autoformer.sh
bash ./scripts/Traffic_script/Autoformer.sh
bash ./scripts/Weather_script/Autoformer.sh
bash ./scripts/ILI_script/Autoformer.sh- Special-designed implementation
-
Speedup Auto-Correlation: We built the Auto-Correlation mechanism as a batch-normalization-style block to make it more memory-access friendly. See the paper for details.
-
Without the position embedding: Since the series-wise connection will inherently keep the sequential information, Autoformer does not need the position embedding, which is different from Transformers.
To easily reproduce the results using Docker, conda and Make, you can follow the next steps:
- Initialize the docker image using:
make init. - Download the datasets using:
make get_dataset. - Run each script in
scripts/usingmake run_module module="bash scripts/ETT_script/Autoformer_ETTm1.sh"for each script. - Alternatively, run all the scripts at once:
for file in `ls scripts`; do make run_module module="bash scripts/$script"; done
See predict.ipynb for workflow (in Chinese).
We experiment on six benchmarks, covering five main-stream applications. We compare our model with ten baselines, including Informer, N-BEATS, etc. Generally, for the long-term forecasting setting, Autoformer achieves SOTA, with a 38% relative improvement over previous baselines.
We will keep adding series forecasting models to expand this repo:
- Autoformer
- Informer
- Transformer
- Reformer
- LogTrans
- N-BEATS
If you find this repo useful, please cite our paper.
@inproceedings{wu2021autoformer,
title={Autoformer: Decomposition Transformers with {Auto-Correlation} for Long-Term Series Forecasting},
author={Haixu Wu and Jiehui Xu and Jianmin Wang and Mingsheng Long},
booktitle={Advances in Neural Information Processing Systems},
year={2021}
}
If you have any questions or want to use the code, please contact [email protected].
We appreciate the following github repos a lot for their valuable code base or datasets:
https://github.com/zhouhaoyi/Informer2020
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent