The primary motivation behind this repository is to develop and fine-tune autonomous navigation strategies for Duckiebots using cutting-edge reinforcement learning techniques. The goal is to create intelligent agents that can maneuver through complex scenarios while adhering to traffic rules, avoiding obstacles, and following optimal paths. By leveraging the power of reinforcement learning and a deep understanding of the Duckietown environment, we seek to achieve high-performance navigation agents that showcase the potential of AI-driven robotics.
TensorTesok_custom.yaml
Custom map for the simulations.basic_control.py
Basic controller for the Duckiebotmap_perview.mp4
Visualization of the simulation on the map.agents.py
Used algorimts/agents for our modeltraning.py
For training our modeltest.py
Testing our modelwrappers.py
Wrappers provided by gym_duckietown with adjustmentutils.py
For nicer codesetup.py
Dependency installersrc/
gym_duckietown repository with fixestrain_models/
train models, try out with test.pyreport_fin.pdf
Report for the project
Prerequirement: Python 3.8
Install necessary dependencies by these commands:
git clone https://github.com/marton12050/T.T-duckietown
cd T.T-duckietown
pip3 install -e .
The training was made by deafult parameters
--seed
Set seed for training, the default is 123--mode
Set which agent to use[ppo|impala]
, default is ppo--map_name
Set training map, the default is 'small_loop'--max_steps
Set maximum step for the training, default is 1000000--num_cpus
Set number of cpu core to use(min 2), default 3--num_gpus
Set number of gpu to use, default 0--training_name
Checkpoint file name--checkpoint
Previous trained modoel checkpoint path to load
Train the model example:
python3 training.py --seed 321 --mode ppo --map_name loop_empty
Evaluate on our custom map
--seed
Set seed for training, the default is 123--mode
Set which agent to use[ppo|impala]
, default is ppo--map_name
Set training map, the default is 'TensorTesok_custom'--max_steps
Set maximum step for the training, default is 1000000--checkpoint
Load trained model(checkpoint path)
python3 test.py --checkpoint <path_to_checkpoint>
For training we used the Ray reinforcement learning libary(RLlib) and Ray Tune for optimization and hyperparameter(Ray Tune) tuning
Used algorithm:
- Impala(Importance Weighted Actor Learner Architecture)
- Impala is an off-policy actor-critic framework that decouples acting from learning and learns from experience trajectories using V-trace.
- PPO(
Proximal Policy Optimization)
- PPO, is a policy gradient method for reinforcement learning. The motivation was to have an algorithm with the data efficiency and reliable performance of TRPO, while using only first-order optimization.
Mainly modified from original wrappers are the croppig and reward functions.
- Rewards
- Rewards are 0 if the agents do somthing invalid (drive in left lane, not on the road...)
- Otherwise, reward calculated based on the travel distance, middle of the right lane distane, curve to the middle of right lane, and the speed
reward = W1*travel_distance+W2*distance_middle+W3*curve_middle
- Cropping
- Crop the sky out(top third of the image) since it has no useful feature