How to run inference/convert_composer_to_hf.py with MPT-1B model on Habana Gaudi 2, file formats do not match about llm-foundry HOT 4 CLOSED

Hi, Gaudi support is alpha, and uses Deepspeed in FSDP. Most of the LLM Foundry repo is built around using FSDP, in particular that helper script assumes a single checkpoint file, which is different from what Deepspeed produces. Unfortunately we don't have an easy script for converting from a Deepspeed checkpoint into another format.

from llm-foundry.

Hi Daniel, thanks for the response, and to be clear, I am an Intel Gaudi employee. My goal here is to take what was documented in the blog: https://www.databricks.com/blog/llm-training-and-inference-intel-gaudi2-ai-accelerators and provide the specific
instructions to run the MPT-1B model with 8 Gaudi cards.

Note that the mpt-1b-gaudi2.yaml that is on your github page has FSDP commented out for Gaudi usage, it's not being used.

So how is the blog executing the commands? You show how to run 8 Gaudi Inference using Hugging Face, and it seems like you have to run that convert_composer_to_hf.py to get to Optimum Habana..

from llm-foundry.

we can close this. As the support is early, we'll stay focused on the training section only.

from llm-foundry.

closed

from llm-foundry.