Comments (2)
Describe the bug Very slow inference during agent work in comparison to usual LLM interaction I'm using local setup with API connection to TextGen WebUI in local network Each iteration of TaskWeaver is very-very slow generation speed is drastically decreased to around 1-2 t/s (usual speed on same setup 15-20 t/s)
At this communication rate this tool is net very useful, simple coding task like print numbers executed in 20-30 mins. Is there any tweak to solve it. I guess it could because of relatively large context in each request?
To Reproduce Steps to reproduce the behavior:
- Start the service
- Type the user query "any listed query from example description"
- Wait for the response forever
Expected behavior Similar inference speed as Autogen
Environment Information (please complete the following information):
- OS: MacOS
- Python Version 3.11
- LLM that you're using: number of different 7b models
hi bro, how to run with local llm
from taskweaver.
Close inactive issues.
from taskweaver.
Related Issues (20)
- taskweaver HOT 3
- Unknown recipient exception HOT 4
- Call plugin exception HOT 2
- test failed ,PermissionError HOT 2
- Facing issue while using the taskweaver as lib in my project inside a dockerised env. HOT 3
- Facing issue while setting and getting session variables. HOT 5
- Taskweaver with multiple LLMs seems not working HOT 2
- plugin doesn't work well when I save artifact HOT 1
- Set session variables inside a plugin HOT 2
- Taskweaver code for Sagemaker endpoint
- debug error in latest version HOT 1
- Track or count the number of tokens being used HOT 2
- Handling Korean Font Issues in Matplotlib Visualizations with Taskweaver and Docker HOT 2
- Not able to leverage session event handler in my code HOT 4
- Request for Scatter plot default marker modify HOT 2
- Permission Errors and Plugin Execution Failures in Dockerized Environment
- I can't import packages which needs to be installed HOT 1
- Failed loading plugin, 'gbk' codec can't decode? HOT 2
- Not able run Taskweaver with LLM Qwen1.5-72B-Chat HOT 2
- Does Taskweaver works with llama3? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from taskweaver.