fix division issue in learning rate decay computation on Python 2

I didn't find how [] to have a minimum learning rate <p

Python 2 compatibility about tensorflow-mnist-tutorial HOT 4 CLOSED

martin-gorner commented on August 30, 2024 2

Python 2 compatibility

from tensorflow-mnist-tutorial.

Comments (4)

martin-gorner commented on August 30, 2024 1

I didn't find how [] to have a minimum learning rate

Hmm, I hear the ancients knew of a secret way. The knowledge is lost now, only an arcane symbol reached us through time, its meaning shrouded in mystery: +

As in:
learning_rate_with_min = min + tf.train.exponential_decay(...)

;-)

from tensorflow-mnist-tutorial.

kakawait commented on August 30, 2024 1

I was totally focused on API signature and searching on doc that I forget the basic arithmetic... Good catch :)

from tensorflow-mnist-tutorial.

kakawait commented on August 30, 2024

Other possible solution will be to delegate decay rate computation to TensorFlow

global_step = tf.Variable(0, trainable=False)
learning_rate = tf.train.exponential_decay(0.003, global_step, 100, 0.97, staircase=True)
train_op = tf.train.AdamOptimizer(learning_rate).minimize(loss, global_step=global_step)

Some result on training phase (3.0 convolutional)

Epoch: 0001 - cost = 0.317358517 - learning rate = 0.002576
Epoch: 0002 - cost = 0.068440056 - learning rate = 0.002212
Epoch: 0003 - cost = 0.044457262 - learning rate = 0.001843
Epoch: 0004 - cost = 0.030760335 - learning rate = 0.001582
Epoch: 0005 - cost = 0.021672835 - learning rate = 0.001318
Epoch: 0006 - cost = 0.014874582 - learning rate = 0.001132
Epoch: 0007 - cost = 0.012011148 - learning rate = 0.000943
Epoch: 0008 - cost = 0.006284746 - learning rate = 0.000810
Epoch: 0009 - cost = 0.004536705 - learning rate = 0.000674
Epoch: 0010 - cost = 0.003143254 - learning rate = 0.000579
Epoch: 0011 - cost = 0.001789587 - learning rate = 0.000482
Epoch: 0012 - cost = 0.001161188 - learning rate = 0.000414
Epoch: 0013 - cost = 0.000715020 - learning rate = 0.000345
Epoch: 0014 - cost = 0.000526037 - learning rate = 0.000296
Epoch: 0015 - cost = 0.000266390 - learning rate = 0.000247
Epoch: 0016 - cost = 0.000154369 - learning rate = 0.000212
Epoch: 0017 - cost = 0.000115868 - learning rate = 0.000177

Only thing you can't reproduce (I didn't find how, I mean) is to have a minimum learning rate

from tensorflow-mnist-tutorial.

martin-gorner commented on August 30, 2024

Python 2 compatibility tested. Everything OK

from tensorflow-mnist-tutorial.

Recommend Projects