Conjugate Gradient Method in TRPO about mushroom-rl HOT 2 CLOSED

The implementation of the fisher dot product is done following this source:
https://www.telesens.co/2018/06/09/efficiently-computing-the-fisher-vector-product-in-trpo/

Notice that the fisher dot product is also adding damping, for this reason, you will not get the same results with normal matrix-vector multiplications.

However, we did test the behavior of the algorithm in deep neural networks, comparing it with the results of OpenAI baselines.
Both methods compute a very similar value for the fisher dot product and the damping parameter. However, probably due to some numerical issues, we notice that the first updates in the torch implementation tend to drive the solution away from the solution (as you might have noticed). However, by increasing the iterations of the conjugate gradient, we get a direction that is extremely close to the one provided by the OpenAI baselines implementation.

With this increased number of iterations, we get performances that are in line with the baselines ones, as you can see from the benchmark:
https://mushroom-rl-benchmark.readthedocs.io/en/latest/source/benchmarks/actor_critic/mujoco.html
https://mushroom-rl-benchmark.readthedocs.io/en/latest/source/benchmarks/actor_critic/bullet.html#

However, this detailed test was performed a long time ago, and it might be that something has changed in the torch implementation. If you find some discrepancy, let us know

from mushroom-rl.

nb: the damping method is really important in the neural network scenario due to the bad conditioning of the fisher matrix in this scenario.

from mushroom-rl.