PlanT: Explainable Planning Transformers via Object-Level Representations

Paper | Project Page

This repository provides code for the following paper:

• Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata and Andreas Geiger, PlanT: Explainable Planning Transformers via Object-Level Representations, CoRL 2022.

Content

• ToDos
• Setup
• Data and models
• Evaluation
• Citation

ToDos

• Best checkpoint + evaluation
• Other checkpoints
• Dataset and training
• Data generation
• PlanT with perception
• Explainability metric

Setup

First, you have to install carla and the conda environment.

# 1. Clone this repository
git clone https://github.com/autonomousvision/plant.git
cd plant
# 2. Setup Carla
# if you have carla already installed, skip the next step AND
# adapt the carla path in setup_env.sh before executing step 3.
chmod +x setup_carla.sh
./setup_carla.sh
# 3. Setup conda environment
chmod +x setup_env.sh
./setup_env.sh

Data and models

You can download our pretrained PlanT-medium 3x by executing:

chmod +x download.sh
./download.sh

The other checkpoints and the dataset used for training will be uploaded soon.

Evaluation

Start a Carla server:

# with display
./carla/CarlaUE4.sh --world-port=2000 -opengl

# without display
SDL_VIDEODRIVER=offscreen SDL_HINT_CUDA_DEVICE=0 ./carla/CarlaUE4.sh --world-port=2000 -opengl

When the server is running, start the evaluation with:

python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTmedium3x eval=longest6

You can find the results of the evaluation in a newly created evaluation folder inside the model folder. If you want to have a (very minimalistic) visualization you can set the debug flag (i.e., python leaderboard/scripts/run_evaluation.py user=$USER experiments=PlanTmedium3x eval=longest6 debug=1)

Citation

If you use this code and data, please cite the following:

@inproceedings{Renz2022CORL,
    author       = {Katrin Renz and Kashyap Chitta and Otniel-Bogdan Mercea and A. Sophia Koepke and Zeynep Akata and Andreas Geiger},
    title        = {PlanT: Explainable Planning Transformers via Object-Level Representations},
    booktitle    = {Conference on Robotic Learning (CoRL)},
    year         = {2022}
}

Also, check out the code for other recent work on CARLA from our group:

• Chitta et al., TransFuser: Imitation with Transformer-Based Sensor Fusion for Autonomous Driving (PAMI 2022)
• Hanselmann et al., KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients (ECCV 2022)
• Chitta et al., NEAT: Neural Attention Fields for End-to-End Autonomous Driving (ICCV 2021)