Comments (6)
Cloud Run may not work well here because it does not allow you to configure number of vCPU per service.
It may be better to use raw Knative for it until Google adds that feature.
from gpt-2-cloud-run.
Interesting issue with trying to put K80s on a n1-highcpu-16
:
The number of GPU dies is linked to the number of CPU cores and memory selected for this instance. For the current configuration, you can select no fewer than 2 GPU dies of this type
So T4 it is.
from gpt-2-cloud-run.
Better solution; actually leverage Python's async to minimize dedicated resources needed, so we can actually use K80s.
With gpt-2-simple
, the generation is done completely in the GPU, so that might work. We might be able to get away with a 4 vCPU n1-standard-4
system (1 vCPU per Pod), and use a K80 (but still 4 concurrent users per Pod, 16 users per Node). The total cost is less than half of what was proposed.
And since it would be 1 vCPU used, we could set up Cloud Run with it, which might be easier than working with Knative.
from gpt-2-cloud-run.
Unfortunately, this is not as easy expected since a tf.Session
cannot be shared between threads and processes, therefore dramatically reducing the async possibilities.
For the initial release I might be OK without, especially if the GPU has high enough throughput.
from gpt-2-cloud-run.
Update: you can share a tf.Session
, but it's not easy and might not necessarily result in a performance gain. It however saves GPU vRAM, which is a necessary precondition. (estimate 2.5GB ceiling when generating 4 predictions at a time, so 4 containers will fit in a 12GB vRAM GPU).
Best architecture is still a 4vCPU + 1GPU w/ 4 containers, but it may be better to see if Cloud Run can assign each container 4vCPUs and then share threads (as Flask's native server is threaded by default and route accordingly). And then see if it causes any deadlocks.
from gpt-2-cloud-run.
Hi Max! Thank you so much for creating gpt-2-cloud-run. It's been really useful and inspiring for my GPT-2 webapp. For this webapp I'm trying to deploy a finetuned 345M GPT-2 Model (~1.4 GB) through Cloud Run on GKE but I am unsure about the spec of the GKE Cluster as well as what concurrency should I set.
Can you please advice on the number of nodes, machine type and concurrency I should be using for maximum cost effectiveness? Currently, I have a concurrency of 1 along with just 1 node (n1-standard-2; 7.5GB; 2vCPU) and a K80 attached to that node but I'm not sure if this is the most cost-effective GKE spec.
I would really appreciate any insights on this! If it helps I intend to deploy only this model and don't plan on having any more GPT-2 webapps.
from gpt-2-cloud-run.
Related Issues (20)
- Add reCAPTCHA support
- Tensorflow.contrib module error when running docker file HOT 1
- Error message on Cloud Run deployment HOT 3
- Is it possible to update this for the big model? HOT 1
- AttributeError: 'NoneType' object has no attribute 'dumps' HOT 1
- Parameter Usage Question
- Poor quality of text generation in Cloud Run compared to Colab HOT 1
- Could not load dynamic library 'libcuda.so.1' HOT 4
- tensorflow version HOT 1
- Dockerfile checkpoint is missing HOT 2
- Concerning generating multiple responses
- Memory limit exceeded HOT 1
- TypeError: not all arguments converted during string formatting
- Google Cloud Free Tier
- Cloud Build workflow could be less janky
- Reduce memory consumption to prevent errors due to container OOM HOT 2
- More consistent output for Save Image
- Save Image button does not always work on Mobile
- Add Favicon indicating when text is generating
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gpt-2-cloud-run.