I'm working with a pretty big model (GPTNeo1.3B) and the gradient attribution features

Just worked it out <a class="user-mention notranslate" data-hovercard-type="user" data

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Support for Distributing Model on Multiple GPU's about ecco HOT 3 CLOSED

jalammar commented on July 25, 2024

Support for Distributing Model on Multiple GPU's

from ecco.

BiEchi commented on July 25, 2024 1

Just worked it out @chrispan68 :

Set device_map=True in from_pretrained() from HuggingFace (in script ecco/src/ecco/__init__.py).
Set gpu=False in from_pretrained() from Ecco (in your own script that uses ecco), to avoid moving the model and data to a specific device (see huggingface/transformers#25145).

Then you can run Ecco with multiple GPUs just like using huggingface.

from ecco.

jalammar commented on July 25, 2024

No similar feature exists in Ecco currently. A PR is welcomed!

from ecco.

BiEchi commented on July 25, 2024

@chrispan68 To fit your model to limited GPUs, there are mainly two ways to go:

Modify the ecco source code to save only attribution features you would like to gain.
Implement multi-GPU implementation, which would require gradient backprop across several GPUs on a single node.

I'm also looking into these two approaches. We can have a discussion if you're still interested.

from ecco.