This repo introduces StreamingFlow (CVPR2024 poster(hightlight)).

Occupancy forecasting on nuScenes dataset

Occupancy forecasting on Lyft dataset

Streaming forecasting: foreseeing the future to 8s

Streaming forecasting: predicting at given interval 0.05s/0.10s/0.25s

We implement StreamingFlow on Vidar codebase and generates streaming prediction on self-supervised 4d occupancy forecasting task with future point clouds as proxy. It is still in an early stage. We provide demo videos of current process.

Streaming forecasting with interval 0.5s:

Streaming forecasting with interval 0.05s:

StreamingFlow is a streaming occupancy forecasting framework which can input multi-modal asynchronous data streams (possibly with different given frequency) as input, and outputs future instance prediction in a continuous manner.

We follow the ST-P3 setup and bevfusion setup for environoment. For data setup, simply organize nuscenes and lyft dataset in ./data/nuscenes and ./data/lyft.

Train command:

python train.py --config /path/to/config

Test command:

python evaluate.py --checkpoint /path/to/checkpoint

We use streamingflow with variable ode step config and checkpoint to conduct the following experiments.

Test command:

 python evaluate.py --checkpoint /path/to/checkpoint --future-frames N 

here, N is for N * 0.5s future seconds.

Test command:

export PYTHONPATH=/project_root_dir/nuscenes-devkit/python-sdk:$PYTHONPATH
python evaluate_streaming.py --checkpoint /path/to/checkpoint --eval-interval N 

here, N is for N * 0.05s interval.

Test command:

python evaluate_datastream.py --checkpoint /path/to/checkpoint --frame-skip N 

here, N is for 20/N interval for lidar input stream interval.

All assets and code are under the Apache 2.0 license unless specified otherwise.

Please consider citing our paper if the project helps your research with the following BibTex:

@inproceedings{shi2024streamingflow,
  title={StreamingFlow: Streaming Occupancy Forecasting with Asynchronous Multi-modal Data Streams via Neural Ordinary Differential Equation},
  author={Shi, Yining and Jiang, Kun and Wang, Ke and Li, Jiusi and Wang, Yunlong and Yang, Mengmeng and Yang, Diange},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14833--14842},
  year={2024}
}

Thanks to prior excellent open source projects:

• GRU_ODE_Bayes
• MotionNet
• FIERY
• ST-P3
• BEVerse
• StretchBEV
• bevfusion
• Vidar