riccardodv / mirrorrl Goto Github PK

View Code? Open in Web Editor NEW

2.0 2.0 0.0 56.13 MB

Python 3.18% Jupyter Notebook 96.82%

mirrorrl's People

Stargazers

Watchers

mirrorrl's Issues

bug

MirrorRL/rl_tools.py

Lines 56 to 68 in 086a736

 def _rollout(self, render=False): 

 # Generates SARSA type transitions until episode's end 

 obs = self.env.reset() 

 act = self.policy(obs) 

 done = False 

 while not done: 

 if render: 

 self.env.render() 

 nobs, rwd, done, terminal = self.env.step(act) 

 nact = self.policy(obs) 

 yield obs, act, rwd, done, terminal, nobs, nact 

 obs = nobs 

 act = nact

I think is actually nact = self.policy(nobs). Do you agree?

unclear code part

MirrorRL/cascade_nn.py

Line 39 in 086a736

 new_neurone.f[0].bias.data = -torch.mean(new_neurone.f[0].weight @ features.t(), dim=1).detach() # I DONT UNDERSTAND; could be related to paper? 

I agree, it is not clear to me this line of the code, not sure if it is in the paper.

SARSA or Fitted-Q iteration or DQN or LSTD

MirrorRL/cascade_mirror_rl_fqi.py

Lines 69 to 72 in b783039

 with torch.no_grad(): 

 newqsp = cascade_qfunc.forward_from_old_cascade_features(nobs_feat).gather(dim=1, index=nact) 

 q_target = rwd + gamma * (nobs_q + newqsp) * not_terminal 

 data_loader = DataLoader(TensorDataset(obs_feat, act, obs_q, q_target), batch_size=batch_size, shuffle=True, drop_last=True)

It seems like this piece of code corresponds more to SARSA method as we use next actions in the trajectory in order to compute new target q-value. Do we have reasons to use SARSA as a way to fit approximation of Q?

Should we try other approaches such as Fitted Q iteration, LSTD or DQN? Do you know what to choose in which situation?

riccardodv / mirrorrl Goto Github PK

mirrorrl's People

Stargazers

Watchers

mirrorrl's Issues

bug

unclear code part

SARSA or Fitted-Q iteration or DQN or LSTD

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def _rollout(self, render=False):
	# Generates SARSA type transitions until episode's end
	obs = self.env.reset()
	act = self.policy(obs)
	done = False
	while not done:
	if render:
	self.env.render()
	nobs, rwd, done, terminal = self.env.step(act)
	nact = self.policy(obs)
	yield obs, act, rwd, done, terminal, nobs, nact
	obs = nobs
	act = nact

	with torch.no_grad():
	newqsp = cascade_qfunc.forward_from_old_cascade_features(nobs_feat).gather(dim=1, index=nact)
	q_target = rwd + gamma * (nobs_q + newqsp) * not_terminal
	data_loader = DataLoader(TensorDataset(obs_feat, act, obs_q, q_target), batch_size=batch_size, shuffle=True, drop_last=True)