mexiqq / llama-70b-chat-4-shards Goto Github PK
View Code? Open in Web Editor NEWllama-70b-chat-4-shards implementation
llama-70b-chat-4-shards implementation
Hi MexiQQ(ๅๅ), Thanks for this great implementation!
But it still could not resolve my concern as though it's 4 shards now, my VRAM for one node(4 gpus)could not handle 70B and I probably need multi-node inference.
DO you have any suggestions or scripts about doing multi-node inference?
When running with the following command on a 4GPU A6000 machine
python convert.py \ --input_llama_path ~/llama/llama-2-70b-chat \ --input_shards 8 \ --output_llama_path ~/llama/llama-2-70b-chat-4-shards \ --ou> --output_shards 4
the following error appear.
Fetching all parameters from the checkpoint at /home/yizzhan/llama/llama-2-70b-chat.
Traceback (most recent call last):
File "/home/yizzhan/miniconda3/envs/llama/lib/python3.8/site-packages/torch/serialization.py", line 619, in save
_save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
File "/home/yizzhan/miniconda3/envs/llama/lib/python3.8/site-packages/torch/serialization.py", line 853, in _save
zip_file.write_record(name, storage.data_ptr(), num_bytes)
RuntimeError: [enforce fail at inline_container.cc:588] . PytorchStreamWriter failed writing file data/338: file write failed
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "convert.py", line 254, in
main()
File "convert.py", line 246, in main
convert_to_llama_70b_2(
File "convert.py", line 219, in convert_to_llama_70b_2
torch.save(shard, path)
File "/home/yizzhan/miniconda3/envs/llama/lib/python3.8/site-packages/torch/serialization.py", line 620, in save
return
File "/home/yizzhan/miniconda3/envs/llama/lib/python3.8/site-packages/torch/serialization.py", line 466, in exit
self.file_like.write_end_of_file()
RuntimeError: [enforce fail at inline_container.cc:424] . unexpected pos 12609115648 vs 12609115544
Hi mexiQQ, thank you for open-sourcing this incredible script. Does this also support llama3?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.