Continuous control from pixels? about mushroom-rl HOT 3 CLOSED

@RylanSchaeffer well, in theory, every deep actor-critic algorithm is already "broken" in the theoretical sense (most notably, the objective is maximizing the undiscounted state distribution instead of the discounted one).

However, if you share the weights, then the objective you are maximizing is not anymore the one presented in the papers. Plus, if you have a KL bound and you change arbitrarily your feature network, you may violate this KL bound (even in expectation!) if you are not careful enough in the implementation.

For this reason, we present the basic versions of the algorithm, as clean as possible and as sound as possible. Implementing a version that shares weights starting from our implementation should be pretty trivial, and we leave these practical use cases to the users.

Furthermore, in my research, I'm currently not much interested in learning continuous actions from pixels. As mushroom is a research tool, we focus principally on methods matching our research or general algorithms.

from mushroom-rl.

In theory, yes. We just never tried.
In practice, results may be worse than with other libraries, which allows sharing the network between the actor and the critic and update both simultaneously.
We will never implement network sharing, as it breaks most reinforcement learning algorithms. However, you can easily implement your favorite actor-critic with a mixed loss between value function target and policy target, exploring already existing implementations.

from mushroom-rl.

@boris-il-forte why does sharing (some) network between the actor and the critic break most RL algorithms?

from mushroom-rl.