pierreexeter / rl_reach Goto Github PK

I realized that my action step was too big and the robot was stuttering a lot.
So I reduced the action step and now I am not able to reach the goal within the maximum time steps (100).
I assume that my options would be to retrain the model with either a larger maximum time steps or to enlarge the action space.

I was wondering if I can scale the action space (or rather the step size) during evaluation without the need to retrain.

Thanks in advance for your help.

Running the function run_experiments.py with the flag alg:her gives me the following error message:

python run_experiments.py --exp-id 96 --algo her --env widowx_reacher-v2 --n-timesteps 10000 --n-seeds 2
Exp #96: Training env widowx_reacher-v2 with algo her and seed 0...
Traceback (most recent call last):
File "train.py", line 272, in
model = exp_manager.setup_experiment()
File "/home/stefan/Documents/Masterarbeit/rl_reach/code/utils/exp_manager.py", line 159, in setup_experiment
env = self.create_envs(self.n_envs, no_log=False)
File "/home/stefan/Documents/Masterarbeit/rl_reach/code/utils/exp_manager.py", line 574, in create_envs
env = ObsDictWrapper(env)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/stable_baselines3/common/vec_env/obs_dict_wrapper.py", line 21, in init
self.spaces = list(venv.observation_space.spaces.values())
AttributeError: 'Box' object has no attribute 'spaces'
Exp #96: Training env widowx_reacher-v2 with algo her and seed 1...
Traceback (most recent call last):
File "train.py", line 272, in
model = exp_manager.setup_experiment()
File "/home/stefan/Documents/Masterarbeit/rl_reach/code/utils/exp_manager.py", line 159, in setup_experiment
env = self.create_envs(self.n_envs, no_log=False)
File "/home/stefan/Documents/Masterarbeit/rl_reach/code/utils/exp_manager.py", line 574, in create_envs
env = ObsDictWrapper(env)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/stable_baselines3/common/vec_env/obs_dict_wrapper.py", line 21, in init
self.spaces = list(venv.observation_space.spaces.values())
AttributeError: 'Box' object has no attribute 'spaces'

I suspect it must have something to do with the observation space being a dictionary.
Maybe you can give me a hint on how to resolve this issue.
Thanks in advance

'evaluate_policy.py' works fine without the --log-info flag.
However, when attempting to create the log info, I get the following error:

Traceback (most recent call last):
File "scripts/plot_episode_eval_log.py", line 17, in
FILE_PATH = str(list(Path(LOG_DIR).rglob('res_episode_1.csv'))[0])
IndexError: list index out of range

The file 'res_episode.csv' does not exist in the logs directory and hence FILE_PATH is empty.
I had trouble finding the script that creates the 'res_episode_1.csv' file. Could you point me in the right direction?
Thanks in advance for your help.

I was wondering whether you have tried to train a model with ddpg + her.
I had some success training sac + her but with ddpg my arm "folds" itself by eventually setting q to joint limits.

If you have, maybe you could share some thoughts on it. Thanks in advance.

When attempting to evaluate the my her-td3 model I get the following error:

(rl_reach) stefan@stefanwanckel:~/Documents/Masterarbeit/rl_reach_ur5e/code$ python evaluate_policy.py --exp-id 11 --n-eval-steps 100 --log-info 1 --plot-dim 2 --render 1
CURR DIRECTORY: /home/stefan/Documents/Masterarbeit/rl_reach_ur5e/code
CURR DIRECTORY: /home/stefan/Documents/Masterarbeit/rl_reach_ur5e/code
Evaluating env ur5e_reacher-v1 with algo her and seed 0...
Traceback (most recent call last):
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/gym/envs/registration.py", line 121, in spec
return self.env_specs[id]
KeyError: 'ur5e_reacher-v1'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "enjoy.py", line 233, in
env_kwargs=env_kwargs,
File "/home/stefan/Documents/Masterarbeit/rl_reach_ur5e/code/utils/utils.py", line 229, in create_test_env
vec_env_kwargs=vec_env_kwargs,
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/stable_baselines3/common/env_util.py", line 102, in make_vec_env
return vec_env_cls([make_env(i + start_index) for i in range(n_envs)], **vec_env_kwargs)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 25, in init
self.envs = [fn() for fn in env_fns]
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 25, in
self.envs = [fn() for fn in env_fns]
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/stable_baselines3/common/env_util.py", line 77, in _init
env = gym.make(env_id, **env_kwargs)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/gym/envs/registration.py", line 145, in make
return registry.make(id, **kwargs)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/gym/envs/registration.py", line 89, in make
spec = self.spec(path)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/gym/envs/registration.py", line 131, in spec
raise error.UnregisteredEnv('No registered env with id: {}'.format(id))
gym.error.UnregisteredEnv: No registered env with id: ur5e_reacher-v1
Traceback (most recent call last):
File "scripts/plot_training_1seed.py", line 35, in
df = pd.read_csv(log_dir + 'stats.csv')
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 610, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 462, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 819, in init
self._engine = self._make_engine(self.engine)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 1050, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 1867, in init
self._open_handles(src, kwds)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 1368, in _open_handles
storage_options=kwds.get("storage_options", None),
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/common.py", line 647, in get_handle
newline="",
FileNotFoundError: [Errno 2] No such file or directory: 'logs/exp_11/her/ur5e_reacher-v1_1/stats.csv'
The environment specified is missing! Please update gym_envs/widowx_env/envs_list.csv. Exiting...
Traceback (most recent call last):
File "scripts/plot_episode_eval_log.py", line 17, in
FILE_PATH = str(list(Path(LOG_DIR).rglob('res_episode_1.csv'))[0])
IndexError: list index out of range

I suppose that the main error is: KeyError: 'ur5e_reacher-v1' and the remaining errors are followup errors.

Is it necessary to put the environment folder (in my case ur5e_env) containing the environment class, urdf files, etc into site-packages/gym/envs or is there a way to add my environment to the registry from the original folder structure?

I would be happy to get some help. Thanks.

evaluate_policy.py works fine for one seed (--log-info 0).
However, evaluating with more than one seed yields the following error for every seed:

Traceback (most recent call last):
File "scripts/plot_training_1seed.py", line 35, in
df = pd.read_csv(log_dir + 'stats.csv')
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 610, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 462, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 819, in init
self._engine = self._make_engine(self.engine)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 1050, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 1867, in init
self._open_handles(src, kwds)
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/parsers.py", line 1368, in _open_handles
storage_options=kwds.get("storage_options", None),
File "/home/stefan/anaconda3/envs/rl_reach/lib/python3.7/site-packages/pandas/io/common.py", line 647, in get_handle
newline="",
FileNotFoundError: [Errno 2] No such file or directory: 'logs/exp_999/ppo/widowx_reacher-v1_3/stats.csv'

The file 'stats.csv' does not exist in any of the seed folders.
I would be happy about some hints on how to solve this.
Thanks in advance for your help.

I have a question regarding the simulation in pybullet.
in the widowx_env, pybullet never does stepSimulation(). Instead, the positions of the robot are forced into the new postition. Why did you choose this approach? In this manner no real physics simulation happens through pybullet...

Thanks in advance

I am having a problem running 1_test_widowx_env.py which is in rl_reach/code/tests/manual. In particular, env = gym.make('widowx_reacher-v47') is throwing following error env = gym.make('widowx_reacher-v47'). To solve this problem, I tried some solution under name "
error while creating custom gym env : No registered env with id". I checked couple of blogs here. The proposed solution was that we needed to import our custom environment module. We are already doing it by running

import gym_envs
from stable_baselines3.common.env_checker import check_env

Then, I tried another solution from here. It was proposed to see the output of

print(gym_envs.__file__) 
print(gym_envs.register) 
print(gym_envs.configuration)

When I ran these commands, I got following errors

[INPUT] print(gym_envs.__file__) 
[OUTPUT] None
[INPUT] print(gym_envs.register) 
[OUTPUT] AttributeError: module 'gym_envs' has no attribute 'register'

The solution for this problem proposed on the blog was that there was something wrong with importing. I don't know much about it. I checked __init__.py in the folder gym_envs and I found following lines there

register(
    id='widowx_reacher-v47',
    entry_point='gym_envs.widowx_env.widowx_env:WidowxEnv',
    max_episode_steps=100,
    kwargs=kwargs_dicts['widowx_reacher-v47'],
    )

It seems that it should work but I don't know why this is throwing error. Any help will be appreciated. Thank you.