A piece of GPU can train this model? about prefixtuning HOT 4 OPEN

@YahooHu Hi, I met the same issue the first I run this code.
You may follow this hyper parameter given by the author.

• set precision fp16
• change --mid_dim from 800 to 512
• reduce --max_source_length to 512

This setting reduces the memory to about 20G on my A100 card.

from prefixtuning.

@zhaone just found there was change to mid_dim. Thanks.

1. Is this reducing the mid_dim of the MLP right? (reducing pretrained gpt mid_dim does not make sense)
2. In table 2, there is Prefix(0.1%) setting. Does this uses preseqlen = 10 (0.1%) ? or reduced some other part of the model (e.g. reducing mid_dim) ? @XiangLi1999

from prefixtuning.

For XSUM, I think you need a GPU with a larger memory (e.g., > 32GB), otherwise, you need to train for very very long time. For table-to-text, I think it's fine.

from prefixtuning.

For XSUM, I use a Tesla A100 (40GB) to train, but still OOM. How can I solve this? @XiangLi1999

RuntimeError: CUDA out of memory. Tried to allocate 1.20 GiB (GPU 0; 39.59 GiB total capacity; 36.82 GiB already allocated; 261.50 MiB free; 37.90 GiB reserved in total by PyTorch)
Epoch 0: 0%| | 0/12784 [00:01<?, ?it/s]

from prefixtuning.