Comments (3)
lvhan028
commented on May 18, 2024
You can try https://github.com/InternLM/lmdeploy, which is a very efficient and high-throughput LLM deployment toolkit for InternLM, LLaMA, vicuna etc.
from internlm.
LiamTTT
commented on May 18, 2024
Will more general model inference optimization be done in the future?
from internlm.
lvhan028
commented on May 18, 2024
Will more general model inference optimization be done in the future?
Yes, it can be found in LMDeploy
from internlm.
Related Issues (20)
- [QA] Windows11使用bitsandbytes运行InternLM2-chat-7B-4bits量化，大模型精神错乱 HOT 19
- [QA] InternLM2微调时数据的max_token
- internlm2-chat-7b本身支持的token多大？[QA] HOT 1
- [Bug] LMdeploy跑出来的demo有问题 HOT 5
- [Bug] `Unrecognized configuration class Error` returned by `AutoTokenizer.from_pretrained` with InternLM-chat-1.8b-sft (transformers==4.36) HOT 2
- 卡住问题和max_position_embeddings[QA] HOT 2
- InternLM2 下载模型后，用本地路径加载tokenizer和模型报错 HOT 2
- [QA] tokenizer对<|im_start|>的特殊编码不对
- [QA] 如果想基于internLM使用领域数据微调出一个领域模型，应该使用internLM-20B-chat还是internLM-20B-sft？ HOT 3
- [Bug] llama.cpp internlm2 function calling bug HOT 6
- [QA] 如何提升推理速度？ HOT 2
- 请问是否支持在MindSpore的910A或者910B上部署？ HOT 5
- [QA] 书生2模型有关chat_template的问题 HOT 5
- [Bug] 微调eval阶段使用generate的结果会出现</s> HOT 2
- 请问是否支持200k上文的微调，需要什么样的配置？[QA] HOT 2
- [QA] InternLM 2 对文字种类的识别, 生成能力以及微调相关问题 HOT 6
- [Bug] InternLM2 int4 出现重复说话、重复前置内容（system prompt）现象 HOT 9
- [Bug] When loading a model by using transformers and using stream chat, it seems no whitespace character in English response. HOT 1
- [QA] Number of training tokens for Internlm2 1.8B, 7B, and 20B? HOT 3
- [QA] 请问是否会开源PPO的训练code和reward model？ HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from internlm.