International Conference on Learning Representation (ICLR), 2024

[Project Page] [Arxiv] [Openreview]

Yinan Zheng*, Jianxiong Li*, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu

The official implementation of FISOR, which represents a pioneering effort in considering hard constraints (Hamilton-Jacobi Reachability) within the safe offline RL setting.

FISOR transforms the original tightly-coupled safety-constrained offline RL problem into three decoupled simple supervised objectives:

• Offline identification of the largest feasible region;
• Optimal advantage learning;
• Optimal policy extraction via time-independent classifier-guided diffusion model, enhancing both performance and stability.

conda create -n env_name python=3.9
conda activate FISOR
git clone https://github.com/ZhengYinan-AIR/FISOR.git
cd FISOR
pip install -r requirements.txt

Run

# OfflineCarButton1Gymnasium-v0
export XLA_PYTHON_CLIENT_PREALLOCATE=False
python launcher/examples/train_offline.py --env_id 0 --config configs/train_config.py:fisor

where env_id serves as an index for the list of environments.

We can run filter_data.py to generate offline data of varying volumes. We also can download the necessary offline datasets (Download link). Then run

python launcher/examples/train_offline.py --env_id 17 --config configs/train_config.py:fisor --ratio 0.1

where ratio refers to the proportion of the processed data to the original dataset.

We need to download the necessary offline dataset for Point Robot environment (Download link). Training FISOR in the Point Robot environment

python launcher/examples/train_offline.py --env_id 29 --config configs/train_config.py:fisor

Then visualize the feasible region by running viz_map.py.

If you find our code and paper can help, please cite our paper as:

@inproceedings{
zheng2024safe,
title={Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model},
author={Yinan Zheng and Jianxiong Li and Dongjie Yu and Yujie Yang and Shengbo Eben Li and Xianyuan Zhan and Jingjing Liu},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=j5JvZCaDM0}
}

Parts of this code are adapted from IDQL and DRPO.