This is the PyTorch implementation for inference and training of the world model and driving policy as described in:

Model-Based Imitation Learning for Urban Driving

Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zak Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla and Jamie Shotton.

NeurIPS 2022
Blog post

Our model can drive in the simulator with a driving plan predicted entirely from imagination.
From left to right we visualise: RGB input, ground truth bird's-eye view semantic segmentation, predicted bird's-eye view segmentation.
When the RGB input becomes sepia-coloured, the model is driving in imagination.

If you find our work useful, please consider citing:

@inproceedings{mile2022,
  title     = {Model-Based Imitation Learning for Urban Driving},
  author    = {Anthony Hu and Gianluca Corrado and Nicolas Griffiths and Zak Murez and Corina Gurau
   and Hudson Yeo and Alex Kendall and Roberto Cipolla and Jamie Shotton},
  booktitle = {Advances in Neural Information Processing Systems ({NeurIPS})},
  year = {2022}
}

• Create the conda environment by running conda env create.
• Download CARLA 0.9.11.
• Install the carla package by running conda activate mile followed by easy_install ${CARLA_ROOT}/PythonAPI/carla/dist/carla-0.9.11-py3.7-linux-x86_64.egg.
• We also need to add ${CARLA_ROOT}/PythonAPI/carla/ to the PYTHONPATH. This can be done by creating a file in the conda environment ~/miniconda3/envs/mile/etc/conda/activate.d/env_vars.sh containing:

#!/bin/bash

export CARLA_ROOT="<path_to_carla_root>"
export PYTHONPATH="${CARLA_ROOT}/PythonAPI/carla/"

• Download the model pre-trained weights.
• Run bash run/evaluate.sh ${CARLA_PATH} ${CHECKPOINT_PATH} ${PORT}, with ${CARLA_PATH} the path to the CARLA .sh executable, ${CHECKPOINT_PATH} the path to the pre-trained weights, and ${PORT} the port to run CARLA (usually 2000).

• Run bash run/data_collect.sh ${CARLA_PATH} ${DATASET_ROOT} ${PORT} ${TEST_SUITE}, with ${CARLA_PATH} the path to the CARLA .sh executable, ${DATASET_ROOT} the path where to save data, ${PORT} the port to run CARLA (usually 2000), and ${TEST_SUITE} the path to the config specifying from which town to collect data (e.g. config/test_suites/lb_data.yaml).

To train the model from scratch:

• Organise the dataset folder as described in DATASET.md.
• Activate the environment with conda activate mile.
• Run python train.py --config mile/configs/mile.yml DATASET.DATAROOT ${DATAROOT}, with ${DATAROOT} the path to the dataset.

Thanks to the authors of End-to-End Urban Driving by Imitating a Reinforcement Learning Coach for providing a gym wrapper around CARLA making it easy to use, as well as an RL expert to collect data.