Discussion: Long Episode Duration about btgym HOT 3 CLOSED

@JacobHanouna ,

Does working with longer episode affect the policy updates frequency?

No, policy updates frequency controlled by rollout_length parameter:
https://github.com/Kismuz/btgym/blob/master/btgym/algorithms/aac.py#L80

Is there a better way to view it or some trick on how to to make it high resolution?

render_size does affect image size, but doesn't affect it's presentation in tensorboard; one possible way to get it bigger is to hack rendering module and use functionality to save image to disk along with passing it to tensorboard (this is builtin backtrader feature; it has been disabled due to execution time concerns):
https://github.com/Kismuz/btgym/blob/master/btgym/rendering/plotter.py#L26

from btgym.

@Kismuz,

policy updates frequency controlled by rollout_length parameter

Can you hypothesizes what would be the impact of learning when choosing different rollout_length values? I'm trying to understand how to best use the long 1-day 'data as info'? or maybe with some form of 'long term auxiliary' reward?

from btgym.

@JacobHanouna ,
The impact is two-fold: first is related to bellman update and value estimation and second about algorithm implementation:

1. Theoretically bellman update should be insensitive to rollout length; in practice it does - should be at least several (4-5) steps to get real reward until we append recursive tail value based on our current V estimation;

2. Policy update frequency: we run SGD step once rollout is finished. If we set rollout about 100 lngth - our estimate of V would be very accurate but convergence (optimisation ) speed will be at least 10x slower vs 10 step rollout. Really even worse considering the fact that in 10 step rollout experiences starting form 11 will be collected under already updated policy (and improved) after first roll, and so on; while in longer case one hundred experiences will be collected under same policy.

3. Estimator implementation. When we use RNN-based estimators on should remember than we actually (and silently) use truncated version of BPTT (we pass rnn state from rollout to rollout but do not allow backward gradient flow between rolls) winch is known to be biased: it favors short-length dependencies in expense of long-range ones.

Roll of size 5 if ok for atari games but can be insufficient to learn more complex patterns. 20 is a good starting point.

from btgym.