We want to create a cluster of servers and train TensorFlow models in a distributed fashion on that cluster.
- catch Ctrl + C Command to terminate ParamServer process
./templates/00_mnist_replica.py
older version of distributed TF./templates/00_between_graph_replication_async_mnist.py
online deploy guide of distributed TF (from: Feb 2019)
compare training time for 1000 global steps
- 2 Param Servers, 2 Workers, op_parallelism_threads=os.cpu.count():
5 min 02 sec
Explanation: Parameter synchronization over WiFi slows down training process (15Mbit network traffic at peak) - 1 Param Servers, 2 Workers, op_parallelism_threads=os.cpu.count():
2 min 42 sec
- 1 Param Servers, 1 Workers (both on i7-2600k), op_parallelism_threads=8:
3 min 02 sec
- 1 Param Servers, 1 Workers (both on i5-MacBook Pro 2013), op_parallelism_threads=4:
5 min 55 sec
- use tf.version >= 1.12 (with older tf versions I received errors during distributed execution)
- use LAN network, avoid Wifi