Hello, I have some questions about running memory about stylegestures HOT 3 OPEN

Hello @StacyIssac,

I don't believe I can answer this myself, but perhaps other people here who have successfully trained these models can? (Or @simonalexanderson may chime in when he's back at work next week.) I'm speculating here, but it might perhaps be useful for other knowledgeable people if you were to post additional information such as which dataset you are running on, what the model size is, and what batch size and optimiser you are using.

For the record, nvidia-smi displays 11019 MiB of memory for each GPU on the server where our models were trained, and I faintly recall us running our models fairly close to the maximum in terms what memory permitted. I don't know if this is the specific kind of information you are looking for, however.

from stylegestures.

Hello @StacyIssac,

I don't believe I can answer this myself, but perhaps other people here who have successfully trained these models can? (Or @simonalexanderson may chime in when he's back at work next week.) I'm speculating here, but it might perhaps be useful for other knowledgeable people if you were to post additional information such as which dataset you are running on, what the model size is, and what batch size and optimiser you are using.

For the record, nvidia-smi displays 11019 MiB of memory for each GPU on the server where our models were trained, and I faintly recall us running our models fairly close to the maximum in terms what memory permitted. I don't know if this is the specific kind of information you are looking for, however.

Hello, I'm sorry to disturb you again. We run into the following problems. If you have encountered similar problems, can you answer them for us?

Traceback (most recent call last):
File "D:\ProgramData\Anaconda\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "train_moglow.py", line 53, in
trainer.train()
File "C:\Users\334\Desktop\StyleGestures-master\glow\trainer.py", line 155, in train
for i_batch, batch in enumerate(progress):
File "D:\ProgramData\Anaconda\lib\site-packages\tqdm\std.py", line 1129, in iter
for obj in iterable:
File "D:\ProgramData\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 279, in iter
exitcode = _main(fd, parent_sentinel)
File "D:\ProgramData\Anaconda\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
MemoryError
return _MultiProcessingDataLoaderIter(self)
File "D:\ProgramData\Anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 719, in init
w.start()
File "D:\ProgramData\Anaconda\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "D:\ProgramData\Anaconda\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\ProgramData\Anaconda\lib\multiprocessing\context.py", line 326, in _Popen
return Popen(process_obj)
File "D:\ProgramData\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "D:\ProgramData\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
OSError: [Errno 22] Invalid argument

from stylegestures.

Anyone who has problems with memory consumption may also wish to read issue #14 and its comments for ways that some memory issues may be overcome.

from stylegestures.