Comments (6)
Great to hear. Closing the issue, but please reopen (or create a new issue) if you see this behavior return. I will get this merged into the main
branch and it will be part of the next MII release.
from deepspeed-mii.
遇到了同样的问题。
from deepspeed-mii.
@weiqisun I have a fix in #350. If you would like to try that branch before we merge: pip install git+https://github.com/Microsoft/DeepSpeed-MII@mrwyattii/fix-return-error
from deepspeed-mii.
Thanks @mrwyattii! However, I'm still seeing this error message. I confirm I installed the module from your branch since I have the updated _invoke_async
function now in the installed lib file.
from deepspeed-mii.
Hmm, I'm not able to reproduce this with the fix I have in #350. Could you try adding a print statement that shows the contents of response? Please add
print(f"RANK {self.inference_pipeline.local_rank} RESPONSE:", [r.to_msg_dict() for r in responses])
just before the return
statement here:
You will want to modify this file on your local system: /home/dyheal1/mambaforge/envs/test/lib/python3.10/site-packages/mii/grpc_related/modelresponse_server.py
Share the output of that print statement. Thanks!
from deepspeed-mii.
Actually, nvm. With a clean setup from scratch, now the error message is gone! I'm not sure if it was due to a not-properly terminated process. I previously had a server running overnight. I stopped the server this morning before updating the mii module and I still observed the same error message after the update. But then I realized there were two leftover mii processes after I terminated the server by client.terminate_server()
:
dyheal1 291962 0.9 0.0 41352944 474320 pts/8 Sl 11:26 0:10 /home/dyheal1/mambaforge/envs/test/bin/python -m mii.launch.multi_gpu_server --deployment-name mii-endpoint --load-balancer-port 50050 --restful-gateway-port 51080 --restful-gateway-procs 32 --load-balancer --model-config eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAibWV0YS1sbGFtYS9MbGFtYS0yLTdiLWhmIiwgInRva2VuaXplciI6ICJtZXRhLWxsYW1hL0xsYW1hLTItN2ItaGYiLCAidGFzayI6ICJ0ZXh0LWdlbmVyYXRpb24iLCAidGVuc29yX3BhcmFsbGVsIjogMiwgImluZmVyZW5jZV9lbmdpbmVfY29uZmlnIjogeyJ0ZW5zb3JfcGFyYWxsZWwiOiB7InRwX3NpemUiOiAyfSwgInN0YXRlX21hbmFnZXIiOiB7Im1heF90cmFja2VkX3NlcXVlbmNlcyI6IDIwNDgsICJtYXhfcmFnZ2VkX2JhdGNoX3NpemUiOiA3NjgsICJtYXhfcmFnZ2VkX3NlcXVlbmNlX2NvdW50IjogNTEyLCAibWF4X2NvbnRleHQiOiA4MTkyLCAibWVtb3J5X2NvbmZpZyI6IHsibW9kZSI6ICJyZXNlcnZlIiwgInNpemUiOiAxMDAwMDAwMDAwfSwgIm9mZmxvYWQiOiBmYWxzZX19LCAidG9yY2hfZGlzdF9wb3J0IjogMjk1MDAsICJ6bXFfcG9ydF9udW1iZXIiOiAyNTU1NSwgInJlcGxpY2FfbnVtIjogMSwgInJlcGxpY2FfY29uZmlncyI6IFt7Imhvc3RuYW1lIjogImxvY2FsaG9zdCIsICJ0ZW5zb3JfcGFyYWxsZWxfcG9ydHMiOiBbNTAwNTEsIDUwMDUyXSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiZ3B1X2luZGljZXMiOiBbMCwgMV0sICJ6bXFfcG9ydCI6IDI1NTU1fV0sICJtYXhfbGVuZ3RoIjogNDA5NiwgImFsbF9yYW5rX291dHB1dCI6IGZhbHNlLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9
dyheal1 292647 1.1 0.0 41353024 474248 pts/8 Sl 11:30 0:10 /home/dyheal1/mambaforge/envs/test/bin/python -m mii.launch.multi_gpu_server --deployment-name mii-endpoint --load-balancer-port 50050 --restful-gateway-port 51080 --restful-gateway-procs 32 --load-balancer --model-config eyJtb2RlbF9uYW1lX29yX3BhdGgiOiAibWV0YS1sbGFtYS9MbGFtYS0yLTdiLWhmIiwgInRva2VuaXplciI6ICJtZXRhLWxsYW1hL0xsYW1hLTItN2ItaGYiLCAidGFzayI6ICJ0ZXh0LWdlbmVyYXRpb24iLCAidGVuc29yX3BhcmFsbGVsIjogMiwgImluZmVyZW5jZV9lbmdpbmVfY29uZmlnIjogeyJ0ZW5zb3JfcGFyYWxsZWwiOiB7InRwX3NpemUiOiAyfSwgInN0YXRlX21hbmFnZXIiOiB7Im1heF90cmFja2VkX3NlcXVlbmNlcyI6IDIwNDgsICJtYXhfcmFnZ2VkX2JhdGNoX3NpemUiOiA3NjgsICJtYXhfcmFnZ2VkX3NlcXVlbmNlX2NvdW50IjogNTEyLCAibWF4X2NvbnRleHQiOiA4MTkyLCAibWVtb3J5X2NvbmZpZyI6IHsibW9kZSI6ICJyZXNlcnZlIiwgInNpemUiOiAxMDAwMDAwMDAwfSwgIm9mZmxvYWQiOiBmYWxzZX19LCAidG9yY2hfZGlzdF9wb3J0IjogMjk1MDAsICJ6bXFfcG9ydF9udW1iZXIiOiAyNTU1NSwgInJlcGxpY2FfbnVtIjogMSwgInJlcGxpY2FfY29uZmlncyI6IFt7Imhvc3RuYW1lIjogImxvY2FsaG9zdCIsICJ0ZW5zb3JfcGFyYWxsZWxfcG9ydHMiOiBbNTAwNTEsIDUwMDUyXSwgInRvcmNoX2Rpc3RfcG9ydCI6IDI5NTAwLCAiZ3B1X2luZGljZXMiOiBbMCwgMV0sICJ6bXFfcG9ydCI6IDI1NTU1fV0sICJtYXhfbGVuZ3RoIjogNDA5NiwgImFsbF9yYW5rX291dHB1dCI6IGZhbHNlLCAic3luY19kZWJ1ZyI6IGZhbHNlLCAicHJvZmlsZV9tb2RlbF90aW1lIjogZmFsc2V9
After manually killing these two processes, I started the server again and the error message is gone. Thanks for the fix!
from deepspeed-mii.
Related Issues (20)
- How does GPT2/Bert models utilize continuous batching feature in MII? HOT 1
- Use of dtype in the mii fastgen HOT 1
- Fp6 eta HOT 2
- How to set trust_remote_code=True in pipeline HOT 2
- why all-reduce takes lots of time for mixtral which is quite larger than that of vllm and tensorrt-llm
- When I start server, after loading model, I got an error of 'grpc.aio._call.AioRpcError' HOT 5
- Add support for Gemma models HOT 1
- Speeding up loading in inference checkpoints HOT 2
- Requests.exceptions.ConnectionError: HOT 2
- How to use DeepSpeed-MII to deploy a LLM model from DeepSpeed/Megatron-DeepSpeed trained checkpoints? HOT 2
- MII Example shows that mii is "Slower" than Baseline!
- ValueError: Unsupported model type roberta HOT 2
- Can DeepSpeed-MII inference on multi gpus with only 1 replica? HOT 2
- Kernel execution error with long context length
- Workarounds for pre-Ampere devices HOT 1
- What is the exact meaning of forward tokens?
- Quantization inference HOT 2
- [NEED HELP] Quantization inference HOT 3
- On M3 Pro Macbook having issues with installation HOT 2
- qwen1.5 model Support? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepspeed-mii.