mschrader15 / reinforcement-learning-sumo Goto Github PK

Reinforcement Learning + traffic microsimulation (via SUMO). Uses Ray RLLIB and forces SUMO into the OpenAI Gym Framework

Home Page: https://maxschrader.io/reinforcement-learning-and-sumo

Python 99.40% Shell 0.60%

sumo traffic-simulation reinforcement-learning openai-gym optimization

reinforcement-learning-sumo's Introduction

Reinforcement Learning + SUMO

Applying reinforcement learning to traffic microsimulation (SUMO)

A minimal example is available in the example folder

Very much a WIP

Structure

This project follows the structure of FLOW closely. I only chose to diverge from FLOW because it abstracted the XML creation for SUMO. For me, this repository plugs in to a greater code-base, that turns real-world ITS data into SUMO traffic demand and traffic light operation. Those tools work ultimately by writing to SUMO XML files, and we didn't want to convert to the FLOW pythonic framework.

Results

Full breakdown: https://maxschrader.io/reinforcement-learning-and-sumo

video on Youtube

ES vs. PPO

RL-Controlled Traffic Signals vs. Calibrated Real-World Simulation

Real World Traffic Signals during Simulation Period

RL-Controlled Traffic Signals during Simulation Period

reinforcement-learning-sumo's People

Contributors

Stargazers

Watchers

Forkers

azri41 gioannides ssx15 wkkdhj tonmoy-ts fatcatzf lsm2842035890

reinforcement-learning-sumo's Issues

requirements.txt

Hello

I only run your models with python==3.8 and ray==1.11.0
Please update them in requirements.txt

After 1 day of training, my computer was out of memory with ray so I coud not have final resutls
The configurations of my PC: i9, ram=32gb

ImportError: cannot import name 'get_parameters' from 'rl_sumo.helpers.utils' (/media/ubuntu/diskB/reinforcement-learning-sumo/rl_sumo/helpers/utils.py)

(app39-dev) ubuntu@ubuntu-H8QG6:/media/ubuntu/diskB/reinforcement-learning-sumo$ python train.py

Traceback (most recent call last):
File "/media/ubuntu/diskB/reinforcement-learning-sumo/train.py", line 7, in
from rl_sumo.helpers.preprocessing import execute_preprocessing_tasks
File "/media/ubuntu/diskB/reinforcement-learning-sumo/rl_sumo/helpers/preprocessing.py", line 2, in
import json5 as json
ModuleNotFoundError: No module named 'json5'
(app39-dev) ubuntu@ubuntu-H8QG6:/media/ubuntu/diskB/reinforcement-learning-sumo$ pip install json5
Collecting json5
Downloading json5-0.9.8.tar.gz (22 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: json5
Building wheel for json5 (setup.py) ... done
Created wheel for json5: filename=json5-0.9.8-py2.py3-none-any.whl size=18604 sha256=f121186c35cfd4acedd7579d7a17e6d0c4433f4dd724e99f4307f500dc3e15af
Stored in directory: /home/ubuntu/.cache/pip/wheels/33/13/22/d6429949983cbc014ae883a13af8b3ce949adcc4cf9196a4b9
Successfully built json5
Installing collected packages: json5
Successfully installed json5-0.9.8
(app39-dev) ubuntu@ubuntu-H8QG6:/media/ubuntu/diskB/reinforcement-learning-sumo$ python train.py

Traceback (most recent call last):
File "/media/ubuntu/diskB/reinforcement-learning-sumo/train.py", line 8, in
from rl_sumo.helpers.utils import get_parameters
ImportError: cannot import name 'get_parameters' from 'rl_sumo.helpers.utils' (/media/ubuntu/diskB/reinforcement-learning-sumo/rl_sumo/helpers/utils.py)

TypeError: Cannot instantiate typing.Union

may i know is that a bug or python version incompatible?

(app39-dev) ubuntu@ubuntu-H8QG6:/media/ubuntu/diskB/reinforcement-learning-sumo$ which python
/home/ubuntu/.pyenv/shims/python
(app39-dev) ubuntu@ubuntu-H8QG6:/media/ubuntu/diskB/reinforcement-learning-sumo$ python --version
Python 3.9.13+
(app39-dev) ubuntu@ubuntu-H8QG6:/media/ubuntu/diskB/reinforcement-learning-sumo$ python train.py

Traceback (most recent call last):
File "/media/ubuntu/diskB/reinforcement-learning-sumo/train.py", line 6, in
from rl_sumo.helpers.register_environment import make_create_env
File "/media/ubuntu/diskB/reinforcement-learning-sumo/rl_sumo/helpers/register_environment.py", line 6, in
def make_create_env(env_params, sim_params, version=0) -> Union(str, object):
File "/home/ubuntu/.pyenv/versions/3.9-dev/lib/python3.9/typing.py", line 344, in call
raise TypeError(f"Cannot instantiate {self!r}")
TypeError: Cannot instantiate typing.Union

How to train, run this model

Hello
Thank you for sharing your code.
But Could you show me how to train and test this model?
Thanks

Make code compatible with ray>=1.13.0

You need to have ray<=1.12.0 installed. Ray made large updates to their codebase in 1.13.0.

You are also welcome to find the equivalent function in 1.13.0 and make a PR!

Originally posted by @mschrader15 in #5 (comment)