Incorrect Shape of Baseline in REINFORCE about mushroom-rl HOT 11 CLOSED

Moreover, if you use only 1 episode to train, then the gradient will always be 0. This is because grad_J_episode is computed as:

grad_J_episode = sum_d_log_pi * (J - baseline)

and plugging in the baseline:

grad_J_episode = sum_d_log_pi * (J - J * sum_d_log_pi / sum_d_log_pi) = sum_d_log_pi * (J - J) = 0

Consequently, with this choice of baseline, an agent trained from single episodes will never learn.

from mushroom-rl.

The baseline implementation is correct. This is the elementwise baseline, not the basic one.

Please, refer to the RL literature about this topic:
Marc Peter Deisenroth; Gerhard Neumann; Jan Peters, A Survey on Policy Search for Robotics, 2013.
https://ieeexplore.ieee.org/document/8186816

Also, the fact that the baseline is zero for a single episode trivially comes from the definition of baseline.

from mushroom-rl.

You misread what I wrote. The baseline isn't zero. The gradient is zero.

from mushroom-rl.

Consequently, if you train the agent only one episode at a time, then the gradient is always zero and the agent will never learn even if given infinite data and infinite time.

That cannot be correct.

from mushroom-rl.

Your citation is a book. Can you please be a little more precise?

from mushroom-rl.

The citation is a survey paper.

The fact that the gradient is zero depends exactly on the meaning of the baseline.

from mushroom-rl.

from mushroom-rl.

The normal use case for REINFORCE is to use multiple trajectories to estimate the gradient.

We will put an assert to ensure that the user cannot run the algorithm with a single trajectory.

from mushroom-rl.

from mushroom-rl.

this is also a behavior that we could consider. we will choose the option that minimizes the code.

REINFORCE without baseline and with a single trajectory is basically a useless algorithm, so it's not worth supporting this use case if it makes the code less clean.

from mushroom-rl.

from mushroom-rl.