Comments (7)
Right now the selfplay function only plays 1 game per call.
I seem to recall that we did this because there was some memory leak somewhere. @amj -- do you remember the reason for playing one game / call?
This is a lot faster than loading the model every time we play a game.
Did you get a chance to profile the change? What's the difference?
from minigo.
@Kashomon If I need to reload the model for each game, it takes time to distribute the model across gpus compared with only loading it once.
from minigo.
no it's not a memory leak, but we'd need to change our cluster management/expectations around the pod lifecycle. Actually, now that we check how many games are played at the beginning of selfplay and quit accordingly, this could work.
I think the savings here are the startup costs + the scheduling costs from k8s, times the number of nodes. On 9s that's 5s out of ~300s or ~1%. On 19x it would probably be less, maybe 20s out of 1200?
Also, @weedwind are you loading the model from a local file? Or a gcs bucket?
(Distributed GPUs! What kind of performance numbers are you seeing vs using a single GPU?)
from minigo.
@amj I can run the code both from my local machine with 2 1080 GPUs, and also on Amazon AWS (16 K80). When I say "distribute", I simply mean the startup cost to copy the models to these GPUs.
from minigo.
@amj Using more gpus does not improve the speed, probably because I was training a 9 by 9 board, and loading the model to 16 gpus took more time than 2 gpus.
from minigo.
We used to play 8 or more games in parallel; we dialed it down to 1 for data freshness reasons - you don't want to be generating game data using model generation 17 when generation 18 or 19 is available.
Also, we were playing 9s on CPU, which mean that the overhead was not that big relative to the CPU playout time for 1 game. The balance is skewed towards higher overhead with a 1080 as you can see.
from minigo.
@weedwind right, this code is for the 19x19 pipeline, where a worker plays a game, exits, and a new worker starts up, checks for if a newer models exists, etc.
Are there further issues on this? it looks like you were able to play multiple games without issue? Please re-open if there are further questions
from minigo.
Related Issues (20)
- run concurrent selfplay without bazel HOT 1
- Running minigo with Sabaki GUI HOT 2
- Problem while building tpu-image HOT 3
- Problem in features.stone_features HOT 1
- Onscreen buttons in lw_demo don't toggle (work)
- Minigo not working on Coral accelerator HOT 4
- Add Edge TPU support to C++ engine HOT 1
- Decouple the conv data format from the input feature layout HOT 8
- How strong is the model in kyu/dan? HOT 7
- 000990-cormorant: stderr thread died HOT 1
- Wrong argument passed in minigui/fetch-and-run.sh HOT 1
- How to communicate with engine easily outside stdin HOT 2
- Support for sending board state to the engine via GTP HOT 6
- Looking for 9x9 model files in .minigo file format HOT 7
- Error on Minigo v15(990)
- tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Assign requires shapes of both tensors to match HOT 3
- The setting of num_readouts to get strongest of minigo
- train.sh in cloud tpu
- Minigo training using Coral Dev Board HOT 1
- ./cc/configure_tensorflow.sh HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from minigo.