Loading DeepSeek-Coder-V2-Instruct using transformers fails due to OOM on 8 * A100 (80GB) about deepseek-coder-v2 HOT 5 CLOSED

to use this model on 8 GPUs with VLLM you have to set --tensor-parallel-size 8. I don't know why they set it to 1 in the example, because it literally means to put all tensors on a single GPU

from deepseek-coder-v2.

+1

from deepseek-coder-v2.

same situation on 8*A800

from deepseek-coder-v2.

same situation on 8*A800

from deepseek-coder-v2.

@apotheque - You're right. However, setting the parameter to 8 was also resulting in failures when I raised the issue.

The latest version of vLLM at the time of writing - 0.5.3.post1 - is now able to load the model.

from deepseek-coder-v2.