Comments (5)
Hi @rishin27 - BentoML internally uses async everywhere for better performance, so this might be an internal issue with BentoML on Yatai, that's not related to user code. May I ask what's the BentoML and Yatai version that you are using?
Note that when a Bento is deployed on Yatai, Runners are by default scheduled as their own separate pods, in order to scale separately from the service code. The async code path is used for converting the runner.run
function call into an async RPC call. This might be an issue with how the deployment and runner communication was set up.
from yatai.
Hi @parano Thanks for the info. I get that yatai is build for autoscaling & kubernetes workload magic. But if the user has nowhere marked their code as async, is it right to send it by default to the async code path ?
The general workflow for a data scientist will be that they'll build the model, save it using bento, check if the service is working using 'bentoml serve'... if everything works out will push it to the yatai server for the deployment. But then if yatai adds async magic (which was not intended) and thing start breaking, it'll not be a good UX.
Yatai - v0.3.1-c3dab74
bentoml - v1.0.0a7
from yatai.
Thanks @rishin27, we will look into this issue more. Ideally by design, if the bentoml serve
works locally, it should definitely work on Yatai. Note that Yatai is still in its alpha release so definitely expect some rough edges at the moment.
But if the user has nowhere marked their code as async, is it right to send it by default to the async code path ?
This is actually fairly common in python web frameworks, such as Sanic or FastAPI, where both sync
and async
handlers can be defined by the user, the framework uses async internally. BentoML server uses async even without Yatai. I think the root cause of the issue is likely not about async, but some settings with the distributed runner setup in yatai deployment.
from yatai.
Thanks @parano for your detailed answer.
Do let me know if i can help, happy to contribute.
from yatai.
Hi @rishin27, could you try it again with the latest version of BentoML and Yatai? The issue should be resolved by now.
from yatai.
Related Issues (20)
- fix trans in BentoCloud HOT 1
- Error: kaniko(bentoImageBuildEngine) apt-get install error HOT 1
- Error: get Bento CR bentoes.resources.yatai.ai not found HOT 1
- Will yatai login ldap be supported? HOT 1
- Deploy a "custom Bentoml image" using "yatai-deployment".
- Support labels from values.yaml
- Error: YAML parse error on yatai/templates/deployment.yaml HOT 4
- Installation of yatai-deployment failed to [ERROR registerYataiComponent...] HOT 5
- Provide a way to disable home page cards
- Cannot access web ui HOT 1
- can not start dashboard when running yarn start HOT 3
- Autoscaling LLM HOT 2
- how to deploy with cutom-runner HOT 1
- Does yatai deployment require saving the model to the bentos model store? HOT 2
- Yatai 2.0 Proposal HOT 1
- Support BentoML 1.2 in Yatai HOT 1
- How can I deploy bentoDeployment with both remote runner and embedded runner?
- How can I host the yatai image in a private registry HOT 3
- Yatai memory leak HOT 7
- Request to set the root path of the Dashboard
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from yatai.