error loading "checkpoint_best.pth" model about scrl HOT 2 CLOSED

I guess the file was corrupted due to the multiple processes trying to write on it at the same time. I suggest you try adding the if-statement if comm.is_main_process(): at the top of the line: https://github.com/kakaobrain/scrl/blob/master/trainer/trainer.py#L140

This might be a bug we overlooked. (We always used the last checkpoint instead of the best one since model selection on the test set -note that we didn't use held-out validation set- was kind of cheating.) We would be grateful if you let us know if your problem has been solved by the suggested solution(or submit a PR for us.)

from scrl.

Thank you for your reply and suggestion.
I think so too. The checkpoint file was corrupted due to the multi-gpu & multi-processes (all ranks) trying to write on it at the same time.
I solved the error by modifying the function def _save_checkpoint(self, tag): as below.

def _save_checkpoint(self, tag):
        save_path = f"{self.cfg.save_dir}/checkpoint_" + str(tag) + ".pth"
        state_dict = {
            'tag': str(tag), 
            'epoch': self.cur_epoch,
            'max_eval_score': self.max_eval_score,
            'max_eval_epoch': self.max_eval_epoch,
            }
        for key, target in self._saving_targets.items():
            if self.cfg.fake_checkpoint:
                target = "fake_state_dict"
            else:
                target = utils.unwrap_if_distributed(target)
                target = target.state_dict()
            state_dict[f"{key}_state_dict"] = target
        ##Add ##
        if tag == 'best':
            if torch.distributed.get_rank() == 0:
                torch.save(state_dict, save_path)
            torch.distributed.barrier()
        else:
            torch.save(state_dict, save_path)
        suffix = (C.debug(" (fake_checkpoint)") 
                    if self.cfg.fake_checkpoint else "")
        #######
        return save_path + suffix

We always used the last checkpoint instead of the best one since model selection on the test set -note that we didn't use held-out validation set- was kind of cheating.
--> I think that it is the correct way for reporting the model's performance.

from scrl.