Comments (3)
justheuristic
commented on May 13, 2024
Related: bigscience-workshop/petals#146
from chat.petals.dev.
DarkIlluminatus
commented on May 13, 2024
it seems like the dialogue length is exceeding the maximum limit set in the source code. You might want to consider modifying this value to accommodate longer dialogues. However, do keep in mind that very long dialogues might have other implications on performance and resource usage. Good luck!
from chat.petals.dev.
borzunov
commented on May 13, 2024
Sorry for the long response - we've addressed this issue a few months ago but didn't respond here.
We increased the inference session length limit to 8192 tokens for the latest models like Llama 2 (70B), Stable Beluga 2, Falcon 180B. This should be enough for most cases - Llama 2 is trained on sequences with up to 4096 tokens, so it's unlikely to perform well on sequences much longer than 8192 tokens anyway.
Also, we updated this chat app to gradually increase allocated memory to save resources: we allocate an inference session for 512 tokens, then expand it to 2048 tokens when needed, then expand to 8192.
Let us know if you have other issues!
from chat.petals.dev.
Related Issues (20)
- Site not available. HOT 2
- Is there any difference between the BLOOMZ-176B accessible through WebSocket API and the original BLOOMZ-176B? HOT 4
- 4096 context length HOT 4
- How to remove the installation HOT 4
- Import : import md and most files on GitHub
- Token decode issues with Unicode characters HOT 3
- Stop sequence missed HOT 3
- Fix speed measurements in case of complex Unicode characters
- OSError: meta-llama/Llama-2-70b-hf is not a local folder and is not a valid model identifier HOT 3
- Integrate with Langchain HOT 2
- Integration in Oobabooga webui? HOT 3
- Can I test your http API at http://chat.petals.dev/api1 using curl? HOT 1
- (question)switching from openai and NPU usage on the opi 5? HOT 2
- ModelInfo.__init__() got an unexpected keyword argument 'name' HOT 1
- Cpufeature requirement not allowing backend access of petals HOT 1
- providing a system prompt
- Add YaLM-100B HOT 1
- [Firefox only] SyntaxError: JSON.parse: unexpected character at line 51 column 73 of the JSON data HOT 1
- resume model downloads HOT 2
- Impossible to select part of the text while AI still generates answers
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chat.petals.dev.