Comments (5)
Hrm. I have encountered this issue also on a 3GB GTX 1060. No matter the image size (even, for example, -s 16 16) it will inevitably crash after a few iterations with an out of memory error, similar to the one above. I can get it running on the CPU (although slowly).
I am wondering if there is a memory leak somewhere: once the train / ascend_txt loop begins, I would expect memory utilization to remain approximately stable. I expanded one of the lines in ascend_txt to:
out = synth(z)
mcout = make_cutouts(out)
nmcout = normalize(mcout)
encoded = perceptor.encode_image(nmcout) <- Commenting this out seems to resolve the crash
iii = encoded.float()
Some further experimentation makes me wonder if there is a problem with CLIP/clip/model.py at:
def forward(self, x: torch.Tensor):
x = self.conv1(x) # shape = [*, width, grid, grid]
x = x.reshape(x.shape[0], x.shape[1], -1) # shape = [*, width, grid ** 2]
x = x.permute(0, 2, 1) # shape = [*, grid ** 2, width]
x = torch.cat([self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device), x], dim=1) # shape = [*, grid ** 2 + 1, width]
x = x + self.positional_embedding.to(x.dtype)
x = self.ln_pre(x)
x = x.permute(1, 0, 2) # NLD -> LND
x = self.transformer(x) # <- Commenting this out also resolves the crash
x = x.permute(1, 0, 2) # LND -> NLD
x = self.ln_post(x[:, 0, :])
if self.proj is not None:
x = x @ self.proj
return x
I don't think I understand CUDA or Torch well enough to propose a solution. I tried following the further path that self.transformer(x) calls and added some del statements, but wasn't able to resolve a possible leak.
Any insight you have, nerdyrodent? Thanks for all your work on this really interesting package.
from vqgan-clip.
Good question, and one I'd like to know the answer too as well!
from vqgan-clip.
Good question, and one I'd like to know the answer too as well!
I also want to know how to get a larger resolution under the condition of limited GPU,I have been looking for this answer when I first tried
from vqgan-clip.
With just 3GB VRAM I'd personally use the colab. If you really want to use just 3GB VRAM, try:
python generate.py -p "An apple" -s 64 64 -cuts 4
from vqgan-clip.
Hrm. I have encountered this issue also on a 3GB GTX 1060. No matter the image size (even, for example, -s 16 16) it will inevitably crash after a few iterations with an out of memory error, similar to the one above. I can get it running on the CPU (although slowly).
I am wondering if there is a memory leak somewhere: once the train / ascend_txt loop begins, I would expect memory utilization to remain approximately stable. I expanded one of the lines in ascend_txt to:
out = synth(z) mcout = make_cutouts(out) nmcout = normalize(mcout) encoded = perceptor.encode_image(nmcout) <- Commenting this out seems to resolve the crash iii = encoded.float()
Some further experimentation makes me wonder if there is a problem with CLIP/clip/model.py at:
def forward(self, x: torch.Tensor): x = self.conv1(x) # shape = [*, width, grid, grid] x = x.reshape(x.shape[0], x.shape[1], -1) # shape = [*, width, grid ** 2] x = x.permute(0, 2, 1) # shape = [*, grid ** 2, width] x = torch.cat([self.class_embedding.to(x.dtype) + torch.zeros(x.shape[0], 1, x.shape[-1], dtype=x.dtype, device=x.device), x], dim=1) # shape = [*, grid ** 2 + 1, width] x = x + self.positional_embedding.to(x.dtype) x = self.ln_pre(x) x = x.permute(1, 0, 2) # NLD -> LND x = self.transformer(x) # <- Commenting this out also resolves the crash x = x.permute(1, 0, 2) # LND -> NLD x = self.ln_post(x[:, 0, :]) if self.proj is not None: x = x @ self.proj return x
I don't think I understand CUDA or Torch well enough to propose a solution. I tried following the further path that self.transformer(x) calls and added some del statements, but wasn't able to resolve a possible leak.
Any insight you have, nerdyrodent? Thanks for all your work on this really interesting package.
Please let us know if anyone made progress here.
from vqgan-clip.
Related Issues (20)
- Do we need to train VQGAN and CLIP together? HOT 2
- ImportError: cannot import name 'instantiate_from_config' HOT 1
- zoom.sh and random.sh file issue HOT 1
- RuntimeError: PytorchStreamReader HOT 1
- CUDA out of memory HOT 1
- "nan" losses issue for some small subset of users HOT 10
- Error with python generate.py -p command after updating to the latest version of Anaconda HOT 3
- Image sizes question HOT 1
- Metal Performance Shaders (MPS) Support
- Yaml.scanner.ScannerError: mapping values are not allowed here HOT 1
- Generate.py file not found from tutorial HOT 1
- Running Locally HOT 1
- I'm having an issue trying to get this working on my own pc. HOT 2
- How to use multiple models at once?
- RuntimeError: cusolver error: CUSOLVER_STATUS_INTERNAL_ERROR, when calling `cusolverDnCreate(handle) HOT 2
- **completely unusable colab!!!** ModuleNotFoundError: No module named 'torch._six' HOT 2
- is vqgan+clip colab dead? HOT 1
- clip loss
- Doubts about generating image results
- ModuleNotFoundError: No module named 'torch._six' HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vqgan-clip.