Loss of PG about tianshou HOT 6 CLOSED

Yes, you are right, that's the discounted returns.
Btw, before training the policy, it performs a reward scaling:

from tianshou.

Thanks a lot! :)

from tianshou.

Assume we have a trajectory with 4 timesteps: s0, s1, s2, s3

Once we have a discount factor g (in test_pg.py it is 0.9), we compute the returns as:
G3 = r3
G2 = r2 + r3 * g
G1 = r1 + r2 * g + r3 * g^2
G0 = r0 + r1 * g + r2 * g^2 + r3 * g^3
The compute_episodic_return does this thing, in other words, batch.returns == G.

Tianshou collects different trajectories but stores them chronologically. You can check out the Collector's documentation for more details.

from tianshou.

Thanks! So are all elements of batch.returns the discounted reward of a trajectory, like G0? Or may some elements of batch.returns be the discounted reward of a part of trajectory, like G1,G2,G3?

What's more, I'd like to ask about why using rew-norm? I try to not use rew-norm and find pg will converge much more slowly than using rew-norm. Thanks a lot!

from tianshou.

len(G) == len(r)
If the trajectory has T timesteps, the G_i is computed over r_i, r_{i+1}, r_{i+2}, ..., r_{T-1}
The network training procedure is quite unstable under no-norm input. This is just a code-level trick, many papers do not openly mention it.

from tianshou.

Thanks a lot!

from tianshou.