- Install and use the included Ananconda environment
$ conda env create -f environment.yml
$ conda activate rlif
You'll need to get your own MuJoCo key if you want to use MuJoCo.
- Add this repo directory to your
PYTHONPATHenvironment variable.
export PYTHONPATH="$PYTHONPATH:$(pwd)"
The datasets used in this experiment can be found here and the expert agents can be found here.
For a more detailed breakdown of the experts, please visit this page.
This repository contains code for both value-based and random intervention strategies of RLIF and supports environments in D4RL(https://arxiv.org/abs/2004.07219).
To run the RLIF value-based intervention experiments, use the following command:
python3.8 -m RLIF.examples.train_rlif_main \
--env_name "pen-expert-v1" \
--sparse_env 'AdroitHandPenSparse-v1' \
--dataset_dir 'ENTER DATASET DIR' \
--expert_dir 'ENTER EXPERT DIR' \
--ground_truth_agent_dir 'ENTER GROUND TRUTH AGENT DIR' \
--logging.output_dir './experiment_output'
All available command options can be seen in examples/train_rlif_main.py.
To run the RLIF random intervention experiments, use the following command:
python3.8 -m RLIF.examples.train_rlif_random \
--env_name "pen-expert-v1" \
--sparse_env 'AdroitHandPenSparse-v1' \
--dataset_dir 'ENTER DATASET DIR' \
--expert_dir 'ENTER EXPERT DIR' \
--ground_truth_agent_dir 'ENTER GROUND TRUTH AGENT DIR' \
--logging.output_dir './experiment_output'
All available command options can be seen in examples/train_rlif_random_main.py.
This repository also contains code for DAgger, HD-DAgger, and an IQL baseline that runs in a DAgger-like setting.
To run these experiments, use the following command:
python3.8 -m RLIF.examples.train_dagger_main \
--env_name "pen-expert-v1" \
--sparse_env 'AdroitHandPenSparse-v1' \
--intervention_strategy 'ENTER INTERVENTION STRATEGY' \
--dataset_dir 'ENTER DATASET DIR' \
--expert_dir 'ENTER EXPERT DIR' \
--ground_truth_agent_dir 'ENTER GROUND TRUTH AGENT DIR' \
--logging.output_dir './experiment_output'
All available command options can be seen in examples/train_dagger_main.py.
This codebase logs experiment results to W&B online visualization platform. To log to W&B, you first need to set your W&B API key environment variable:
export WANDB_API_KEY='YOUR W&B API KEY HERE'
Then you can run experiments with W&B logging turned on:
python3.8 -m RLIF.examples.train_rlif_main \
--env_name "pen-expert-v1" \
--sparse_env 'AdroitHandPenSparse-v1' \
--dataset_dir 'ENTER DATASET DIR' \
--expert_dir 'ENTER EXPERT DIR' \
--ground_truth_agent_dir 'ENTER GROUND TRUTH AGENT DIR' \
--logging.output_dir './experiment_output' \
--logging.online
For full working examples, you can run a sweep of DAgger, HG-DAgger, value-based RLIF for Adroit Pen, value-based RLIF for Walker2d, and random intervention RLIF for Hopper using the following commands:
bash scripts/run_pen_dagger.sh
bash scripts/run_pen_hgdagger.sh
bash scripts/run_pen_value_based_rlif.sh
bash scripts/run_walker2d_value_based_rlif.sh
bash scripts/run_hopper_random_rlif.sh
This will generate the plots below.
If you found this code useful, consider citing the following paper:
@misc{luo2023rlif,
title={RLIF: Interactive Imitation Learning as Reinforcement Learning},
author={Jianlan Luo and Perry Dong and Yuexiang Zhai and Yi Ma and Sergey Levine},
year={2023},
eprint={2311.12996},
archivePrefix={arXiv},
primaryClass={cs.AI}
}
The implementation of RLIF builds on the original implementation of RLPD by Ilya Kostrikov and also adapts CQL and IQL from Xinyang Geng's JaxCQL and Ilya Kostrikov's IQL, respectively.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent