kostis-s-z / exploring_meta Goto Github PK

Currently the scripts do not work unless you run the experiments through PyCharm and specify the root folder as Sources root

Quick fix is to install a previous build
pip install git+https://github.com/rlworkgroup/metaworld.git@58546ff25211883ca14d036b3516fe63382c6071#egg=metaworld

MAML-VPG and MAML-PPO seem not to be working well.

MAML-TRPO seems fine.

A difference between these implementations is that the first two use

    learner.adapt(loss)

whereas TRPO uses:

    gradients = torch.autograd.grad(loss, learner.parameters(),
                                    retain_graph=second_order,
                                    create_graph=second_order,
                                    allow_unused=anil)


    learner = l2l.algorithms.maml.maml_update(learner, inner_lr, gradients)

• Add download parameter to mini_imagenet

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

After I modify the path and run the program, the following error occurs. Can someone help me solve it?

When I train my agent, I find that the action always be the -1 or 1 (after clipping)，I wonder how I can solve this problem. Is the lr too large?

• Maybe try Omniglot?
• Is there a way to know its not just a hyperparameter tuning issue?
• Make sure the "features" (the body of the network) does actually change (use CCA withing training)

Has been tested & works as expected

• MAML-PPO
• MAML-TRPO
• ANIL-TRPO
• ANIL-PPO

Make code available to repo to test.

instead of modifying l2l

Currently its Python 3.7 dependent since the model is built through a dictionary that assumes order of elements appended.

1. weights[1:].add_(-1.0, dones[:-1])
   ->
   weights[1:] = dones[:-1] - 1.0

2. p.data.add_(-stepsize, u.data)
   ->
   p.data = u.data + (-stepsize)

        for train_episodes in train_replays:
            new_policy = fast_adapt_trpo_a2c(new_policy, train_episodes, baseline,
                                             fast_lr, gamma, tau, first_order=False, device=device)

->

        for train_episodes in train_replays:
            # Calculate loss & fit the value function
            loss = trpo_a2c_loss(train_episodes, new_policy, baseline, params['gamma'], params['tau'], device)

            # First or Second order derivatives
            gradients = torch.autograd.grad(loss, new_policy.parameters(),
                                            retain_graph=True,  # First order = False
                                            create_graph=True)

            # Perform a MAML update of all the parameters in the model variable using the gradients above
            new_policy = l2l.algorithms.maml.maml_update(new_policy, params['inner_lr'], gradients)

4. Remove MAML module wrapper to in maml_trpo and anil_trpo

My fork of cherry-rl which enables saving success metrics is not compatible with Particles2D