Comments (3)
Hi, @GetUpAt8
Thanks for your question! I regret to tell you that currently our project does not implement the parallel operation of the model on the GPU in the inference stage. We can provide some ideas for implementing model parallel computing in the inference stage. You can use the pipeline to divide each layer into different GPUs in parallel. For implementation, you can refer to Gpipe, or use zero-infinity to load the model to the CPU, and then transfer each layer to the GPU for inference.
Hope it can help you!
from bigtranslate.
Hi, @GetUpAt8
We have implemented parallel inference on multiple GPUs by utilizing tensor_parallel. You can achieve parallel inference through the latest code of the repository.
I wish it could help you!
from bigtranslate.
Thank u so much for your replies and update in parallel inference! I'll try it.
from bigtranslate.
Related Issues (9)
- Translation prompts HOT 1
- max tokens length
- Training and data HOT 2
- 7B model HOT 1
- How to download the translated data? HOT 1
- supported languages HOT 1
- May be we can upgrade to llama2? HOT 1
- 推理不响应指令 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigtranslate.