Git Product home page Git Product logo

Comments (4)

liuqidong07 avatar liuqidong07 commented on July 20, 2024

您好，感谢对我们工作的关注。因为我没有试过zero3训练，所以也无法给出准确的解决方案。但是从报错来看，可能是模型训练完后保存adapter出了问题，建议把--save_steps设置为1，然后调试一下保存checkpoint的代码，具体可以调试一下peft/utils/save_and_load中的set_peft_model_state_dict函数。
祝好

from leader-pytorch.

cwwhh avatar cwwhh commented on July 20, 2024

谢谢您的帮助，但是修改save_steps后还是同样的报错。请问你们用的多少GB的内存呢？或者是否可以将训练好的代码上传一下呢？

from leader-pytorch.

liuqidong07 avatar liuqidong07 commented on July 20, 2024
  1. 您好，因为checkpoint过大，不是很方便上传，我再想一想办法。
  2. 此外，修改save_step并不能直接解决问题，而是需要调试一下存储checkpoint的代码是否存在问题，因为报错显示的是存储的LoRA参数维度是0，建议可以先想办法跑通基础的LoRA代码。
  3. 我理解您想问的是训练时的显存？我们训练时用的是4卡V100 32G。
    祝好

from leader-pytorch.

cwwhh avatar cwwhh commented on July 20, 2024

好的，谢谢您。

from leader-pytorch.

Related Issues (8)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.