Thank you for this awesome contribution, @prigoyal!

Would it be possible to update the tutorial to pytorch-1.0 API? Thank you!

modules is not defined in RNN example. What makes sense would be to use the number of timesteps?

I split my model on two GPUs and use checkpointing to reduce memory usage, but I still got an OOM error when calling backward() , it seem like all the data is proccesed in one of my GPU when calling bakcward()? (I'm using gpu 2 and 3)
Here is my memory usage before backward().

Here is it after backward().

Any advice is appreciated.

Hi,

Thank you for the great repository :)

I have quickly searched and wasn't able to locate the implementation of checkpoint_sequential in the master branch of the pytorch (which is specified as a dependency for your tutorial).
Is it still supposed to be there and I just somehow couldn't find it?

Thank you.

This is probably an issue on my part, but when I run the "Checkpointing_for_PyTorch_models" tutorial, when I get to self.assertEqual(out_checkpointed, out_not_checkpointed), get the following error NameError: name 'self' is not defined. Am I missing something?

Thanks in advance.

I am working with a 0.4.0a0 version and the WLM test is failing in repackage_hidden() function. The error message is copied below. I think it has something to do w/ the fact that the Variable API has been deprecated. Variable functionality works but the type() command returns torch.Tensor and not Variable.
I changed the repackage_hidden() to:
def repackage_hidden(self, h): """Wraps hidden states in new Variables, to detach them from their history.""" #if type(h) == Variable: if type(h) in [Variable, Tensor] : return Variable(h.data) else: return tuple(self.repackage_hidden(v) for v in h)

and it seem to work. Could you please check the issue and validate the fix?

=======================================================================
ERROR: test_wlm_baseline (main.TestMemoryBaseline)

.....
.....
.....
File "test_memory_baseline.py", line 219, in repackage_hidden
return tuple(self.repackage_hidden(v) for v in h)
File "/opt/conda/envs/pytorch-py3.6/lib/python3.6/site-packages/torch/tensor.py", line 351, in iter
raise TypeError('iteration over a 0-d tensor')
TypeError: iteration over a 0-d tensor

Hi Priya,

I was trying to reproduce your benchmarks results. I think two important information are missing. First, what is the GPU that you used to train these models and how much memory does it have? Second, I don't understand how you have computed max image and what it corresponds to. For example in the VNET, does it correspond to N?

Hi Priya,
It is unclear how you run the benchmark. I noticed that for best results I need to run each unit-test independently. Otherwise the first tests seem to "steal" memory from later tests. If the benchmark is intended to run as a whole, probably need to find a way to clean the memory between the tests. Otherwise, maybe need to state clearly in the Readme that the tests are supposed to run independently.
Thanks
Dmitri

Hi,

I found that the "torch.utils.checkpoint module" is not available in pytorch master. Could you provide your implementation in the repo?

Thanks.

It gives 404 error.