Hi Ben

Thanks for the interesting notebooks. Upon studying the "3 - Advantage Actor Critic (A2C) [CartPole].ipynb" notebook, I came to the conclusion that detaching the returns in the update_policy() function is not necessary. The returns are only calculated on the rewards which are environment outputs and therefore not part of the computational graph. So even leaving out the .detach() call should not affect the model. Would you agree?

Hello, thank you for making this repo,
I think while calculating the returns you should take done into consideration as,

    def calculate_returns(self, rewards, dones, normalize = True):
       
        returns = []
        R = 0
        for r, d in zip(reversed(rewards), reversed(dones)):    
            if d:
                R = 0
            R = r + R * self.gamma
            returns.insert(0, R)
            
        returns = torch.tensor(returns).to(device)
        
        if normalize:
            returns = (returns - returns.mean()) / returns.std()
            
        return returns

Also can you please briefly describe the Generalized Advantage Estimation (GAE) while calculating the advantages.

Hello! I've been learning how to code RL form your repo. I've replace duplicating code lines from
def train
def update_policy

to agent's method self.sample_action(). And it seems that agent now solves Cart-Pole problem x2 slower(num of episodes). And it happes everytime. I have no idea what happens with torch and havn't found anything on Internet.
Can you pls help me?

https://github.com/lemikhovalex/pytorch-rl
5_tr - Proximal Policy Optimization (PPO) [CartPole]-Copy1.ipynb

Hello, Ben!
Thank you for a great tutorial series. I have a question regarding your actor-critic notebook.
In function update_policy

def update_policy(returns, log_prob_actions, values, optimizer):

    returns = returns.detach()
    
    policy_loss = - (returns * log_prob_actions).sum()
    
    value_loss = F.smooth_l1_loss(returns, values).sum()
    
    optimizer.zero_grad()
    
    policy_loss.backward()
    value_loss.backward()
    
    optimizer.step()
    
    return policy_loss.item(), value_loss.item()

in policy loss you calculate the usual policy gradient for agent, in value loss you calculate loss for the critic. They seem to be independent, the critic does not affect the agent at all. Shouldn't returns for policy loss be calculated with values from critic or something like that?