google-research / morph-net Goto Github PK
View Code? Open in Web Editor NEWFast & Simple Resource-Constrained Learning of Deep Network Structure
License: Apache License 2.0
Fast & Simple Resource-Constrained Learning of Deep Network Structure
License: Apache License 2.0
Hello. Your work is great.
May I ask for your advice for adapting it to object detection problems?
HI, I have pruned the resnet18 with flops regularizer, but the final structure of pruned resnet18 model is strange. The first conv of 7×7 kernel and 64 filter is pruned to zero channels, so that the network can not run. Do you have any idea to deal with this situation?
Hi,
I'm applying MorphNet on tf.contrib.layers.conv3d(), but it seems that no ops are found.
INFO:tensorflow:OpRegularizerManager starting analysis from: [<tf.Operation 'fc/Relu' type=Relu>].
INFO:tensorflow:OpRegularizerManager found 46 ops and 0 sources.
INFO:tensorflow:OpRegularizerManager regularizing 0 groups.
WARNING:tensorflow:No supported ops found.
Here is my code:
imgs = tf.placeholder(tf.float32, (4, 8, 24, 24, 3))
net = tf.contrib.layers.conv3d(imgs, num_outputs=64, kernel_size=7, name='conv1')
net = tf.nn.max_pool3d(net, [1, 2, 2, 2, 1], [1, 2, 2, 2, 1], padding='VALID', name='maxpool1')
net = tf.contrib.layers.batch_norm(net, center=True, scale=True, is_training=True, scope='bn1')
logits = tf.contrib.layers.fully_connected(net, 10, scope='fc')
network_regularizer = model_size_regularizer.GammaModelSizeRegularizer([logits.op], gamma_threshold=1e-3)
regularization_strength = ....
Does MorphNet support 3d convolution? Thanks!
I am new to Tensorflow, and I want to konw if there is a way to output a pruned model file for future training? Now morph-net output variable name and channel number, but user must construct pruned model manaully every time after pruning
INFO:tensorflow:OpRegularizerManager starting analysis from: [<tf.Operation 'clone_0/semantic' type=Identity>].
INFO:tensorflow:OpRegularizerManager found 8092 ops and 0 sources.
INFO:tensorflow:OpRegularizerManager regularizing 0 groups.
WARNING:tensorflow:No supported ops found.
WARNING:tensorflow:No supported ops found.
Why no supported ops found? Is this correct?
I find it a bit tricky to plug any network into morphnet.. Providing an example with end to end implementation of any small network(say with 3 hidden layers) for some classification problem with BatchNormalization and plugging it with morphnet would be helpful...
There are two types of OpRegularizer
, GammaL1Regularizer
and GroupLassoRegularizer
. The classes GammaFlopsRegularizer
and GroupLassoFlopsRegularizer
in flop_regularizer.py
implements two NetworkRegularizer
targeting FLOPs. Now I have a ResNet model, how can I set regularizer? Does GammaFlopsRegularizer
alone work well? Or shall I use GammaFlopsRegularizer
and GroupLassoFlopsRegularizer
together to train the model?
I use resnet_v1 to train the model, should I let batch_normalization's center=false? I don't know if morphnet has an impact on beta, cause I limit the gamma, but if my beta is very large, it seems not work, I think it's a problem.
Here is my bn parameter below.
tf.layers.batch_normalization( inputs=inputs, axis=1 if data_format == 'channels_first' else 3, momentum=_BATCH_NORM_DECAY, epsilon=_BATCH_NORM_EPSILON, **_center=True_,** scale=True, training=training, fused=FLAGS.enbl_fused_batchnorm)
Does anyone has any results of this approach in comparison with the tensorflow graph optimization approach?
Thanks for your open souring. I'm confused about whether it doesn't work for the full connection layer?
Just wanted to confirm. Once we have trained our initial model we have a json file containing the optimized output sizes of the different layers. When I want to retrain a new model with these newly optimized output sizes I have to create a new model from scratch with these newly optimized output sizes, correct? I've been looking through the documentation to see if there is some kind of function that will do it for us, but haven't been able to find anything.
Hi,thank you very much for this awesome project!
I use morphnet on MobilnetV2 and get errors like:
"RuntimeError: OpRegularizerManager could not handle ops: [u'MobilenetV2/Logits/Dropout/dropout_1/Shape (Shape)', u'MobilenetV2/Logits/AvgPool (AvgPool)']"
How could I solve this error?
I have a situation in which the ITERATION_LIMIT
is even when len(self._all_ops)
is only about 3000 in OpRegularizerManager
. It seems that certain ops keep cycling back on the self._op_deque
. Is there a specific reason this happens? More generally, why are the same ops processed multiple times, even when they have the same type and handler?
In a simple LeNet training example, I go through 34 iterations of assign_grouping when there are far fewer ops to process
Itr: 0 Op: base/conv1/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 1 Op: base/conv1/BatchNorm/gamma/read Type: Identity Handler: grouping_op_handler
Itr: 2 Op: base/conv1/BatchNorm/gamma Type: VariableV2 Handler: grouping_op_handler
Itr: 3 Op: base/conv1/Conv2D Type: Conv2D Handler: output_non_passthrough_op_handler
Itr: 4 Op: Placeholder Type: Placeholder Handler: grouping_op_handler
Itr: 5 Op: base/conv1/Relu Type: Relu Handler: grouping_op_handler
Itr: 6 Op: base/pool1/MaxPool Type: MaxPool Handler: grouping_op_handler
Itr: 7 Op: base/conv2/Conv2D Type: Conv2D Handler: output_non_passthrough_op_handler
Itr: 8 Op: base/conv2/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 9 Op: base/conv2/BatchNorm/gamma/read Type: Identity Handler: grouping_op_handler
Itr: 10 Op: base/conv2/BatchNorm/gamma Type: VariableV2 Handler: grouping_op_handler
Itr: 11 Op: base/conv2/Relu Type: Relu Handler: grouping_op_handler
Itr: 12 Op: base/pool2/MaxPool Type: MaxPool Handler: grouping_op_handler
Itr: 13 Op: base/conv3/Conv2D Type: Conv2D Handler: output_non_passthrough_op_handler
Itr: 14 Op: base/conv3/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 15 Op: base/conv3/BatchNorm/gamma/read Type: Identity Handler: grouping_op_handler
Itr: 16 Op: base/conv3/BatchNorm/gamma Type: VariableV2 Handler: grouping_op_handler
Itr: 17 Op: base/conv3/Relu Type: Relu Handler: grouping_op_handler
Itr: 18 Op: base/conv_output/Conv2D Type: Conv2D Handler: output_non_passthrough_op_handler
Itr: 19 Op: base/conv_output/BiasAdd Type: BiasAdd Handler: grouping_op_handler
Itr: 20 Op: base/conv_output/biases/read Type: Identity Handler: grouping_op_handler
Itr: 21 Op: base/conv_output/biases Type: VariableV2 Handler: grouping_op_handler
Itr: 22 Op: Mean Type: Mean Handler: grouping_op_handler
Itr: 23 Op: base/conv1/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 24 Op: base/conv1/BatchNorm/gamma/read Type: Identity Handler: grouping_op_handler
Itr: 25 Op: base/conv2/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 26 Op: base/conv2/BatchNorm/gamma/read Type: Identity Handler: grouping_op_handler
Itr: 27 Op: base/conv3/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 28 Op: base/conv3/BatchNorm/gamma/read Type: Identity Handler: grouping_op_handler
Itr: 29 Op: base/conv_output/BiasAdd Type: BiasAdd Handler: grouping_op_handler
Itr: 30 Op: base/conv_output/biases/read Type: Identity Handler: grouping_op_handler
Itr: 31 Op: base/conv1/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 32 Op: base/conv2/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 33 Op: base/conv3/BatchNorm/FusedBatchNorm Type: FusedBatchNorm Handler: batch_norm_source_op_handler
Itr: 34 Op: base/conv_output/BiasAdd Type: BiasAdd Handler: grouping_op_handler
I just use the first three bn layers(conv0-bn0-leakyrelu0, conv1-bn1-leakyrelu1, conv2-bn2-leakyrelu2 ) of network that total 74 layers, leakyrelu2 as output.op,input_data as input.op.
when runing have a error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: Retval[0] does not have value
(1)()->None
the morphnet can act on some layers of the model?
Hi
Thanks a lot for open sourcing the morph-net code. It is something i really wish to use since i spent a lot of time trying to get my network to train last month for the RL problem i am working on. I have a model in Keras with multiple losses passed in a dict. Now if i wish to add e.g. a flops-regularizer to my model I wrote something like the following:
state_input = Input(shape=(self.board_size, self.board_size+self.grid_size,
self.stack_frame_size,))
actual_value = Input(shape=(1,))
predicted_value = Input(shape=(1,))
#adv = Input(shape=(1,))
old_prediction_a = Input(shape=(self.output_size,))
old_prediction_b = Input(shape=(self.output_size,))
old_prediction_c = Input(shape=(self.output_size,))
old_prediction_d = Input(shape=(self.output_size,))
c1 = Conv2D(32, 10, strides=9, activation='relu', name='Conv1')(state_input)#6
flat = Flatten()(c1)
d1 = Dense(2048, activation='relu', name='Dense1')(flat)
logits_a = Dense(self.output_size, activation = 'softmax',name='logits_a')(d1)
logits_b = Dense(self.output_size, activation = 'softmax', name='logits_b')(d1)
logits_c = Dense(self.output_size, activation = 'softmax', name='logits_c')(d1)
logits_d = Dense(self.output_size, activation = 'softmax', name='logits_d')(d1)
model = Model(inputs=[state_input, actual_value, predicted_value,
old_prediction_a, old_prediction_b,
old_prediction_d,old_prediction_c],
outputs=[logits_a, logits_b, logits_c, logits_d])
model.load_weights(weights_path)
network_regularizer = flop_regularizer.GammaFlopsRegularizer(
[logits_a.op, logits_b.op, logits_c.op, logits_d.op], gamma_threshold=1e-3)
self.actor_exporter = structure_exporter.StructureExporter(network_regularizer.op_regularizer_manager)
regularization_strength = 1e-10
regularizer_loss = (network_regularizer.get_regularization_term() * regularization_strength)
losses = {
'logits_a': ppoloss(actual_value=actual_value, predicted_value=predicted_value,
old_prediction=old_prediction_a),
'logits_b': ppoloss(actual_value=actual_value, predicted_value=predicted_value,
old_prediction=old_prediction_b),
'logits_c': ppoloss(actual_value=actual_value, predicted_value=predicted_value,
old_prediction=old_prediction_c),
'logits_d': ppoloss(actual_value=actual_value, predicted_value=predicted_value,
old_prediction=old_prediction_d),
}
model.compile(optimizer=Adam(lr=1e-4),
loss=[losses])
`
If i add regularizer loss to my dict i get an error that since my outputs number 4 I must have only 4 losses. I cannot add it to the losses since then i have a problem that losses is a dict and regularization_loss is a float.
How may I add the loss in my particular problem
Hi, thanks for open soure.
I met a problem: the regularizer_loss is nan
the code is resnet50 in tensorflow
the goal is classification of cifar-100
the loss used is GammaFlopsRegularizer
I tried multiple regularization_strength, but it is still nan.
What should I do to solve this problem? Thanks for your answer.
Hi,
I am trying to start working with Morph-Net and can't seem to install the required Python modules. Running the setup.py script generates the expected .egg file and it is copied to the appropriate location. However I can't seem to get the various modules recognised. Not sure if I am making a rookie mistake or missing something here.
Hi, thanks for open souring. when I followed the instructions for extracting the Architecture Learned by MorphNet, the code reported an error.
NotFoundError: xxxxxxx/epoch_1/learned_structure/alive_999; No such file or directory
However, when I see the code following:
def create_file_and_save_alive_counts(self, train_dir: Text,
global_step: tf.Tensor) -> None:
"""Creates a file and saves live counts to it.
Creates the directory {train_dir}/learned_structure/ and saves the current
alive counts to {path}/{_ALIVE_FILENAME}_{global_step} and overwrites
{path}/{_ALIVE_FILENAME}.
Args:
train_dir: where to export the alive counts.
global_step: current value of global step, used as a suffix in filename.
"""
current_filename = '%s_%s' % (_ALIVE_FILENAME, global_step)
directory = os.path.join(train_dir, 'learned_structure')
try:
tf.gfile.MkDir(directory)
except tf.errors.OpError:
# Probably already exists. If not, we'll see the error in the next line.
pass
with tf.gfile.Open(os.path.join(directory, current_filename), 'w') as f:
self.save_alive_counts(f)
with tf.gfile.Open(os.path.join(directory, _ALIVE_FILENAME), 'w') as f:
self.save_alive_counts(f)
the code does not try to make file for os.path.join(directory, current_filename) and os.path.join(directory, _ALIVE_FILENAME).
Is the way I use incorrectly?
thanks for your reply
Hi, Im currently trying to train a simple ConvAE on MNIST, using a GammaFlopsRegularizer
on out.op
of the following network
def conv_ae(x_ph, is_training, scope, channels=[32, 64, 64], reuse=False):
norm_params = {'is_training': is_training, 'scale': True, 'center': False}
# Network Definition
with tf.variable_scope(scope, reuse=reuse):
with slim.arg_scope([slim.conv2d, slim.fully_connected],
normalizer_fn=slim.batch_norm,
normalizer_params=norm_params,
weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
weights_regularizer=slim.l2_regularizer(0.0005)):
conv1 = slim.conv2d(x_ph, num_outputs=channels[0], kernel_size=3, scope='conv1')
pool1 = slim.max_pool2d(conv1, kernel_size=2, scope='pool1')
conv2 = slim.conv2d(pool1, num_outputs=channels[1], kernel_size=3, scope='conv2')
pool2 = slim.max_pool2d(conv2, kernel_size=2, scope='pool2')
conv3 = slim.conv2d(pool2, num_outputs=channels[2], kernel_size=3, scope='conv3')
# deconv part
deconv1 = slim.conv2d_transpose(conv3, num_outputs=channels[2], kernel_size=3, stride=2, scope='conv_transpose1')
deconv2 = slim.conv2d_transpose(deconv1, num_outputs=channels[1], kernel_size=3, stride=2, scope='conv_transpose2')
out = slim.conv2d_transpose(deconv2, num_outputs=1, kernel_size=3, stride=1, activation_fn=tf.nn.sigmoid, scope='conv_output')
return out
However, I'm seeing the OpRegularizerManager
hit the iteration limit and found the following error:
Traceback (most recent call last):
File "deconv_test.py", line 33, in <module>
network_regularizer = flop_regularizer.GammaFlopsRegularizer([pred.op], gamma_threshold=gamma_scale)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/network_regularizers/flop_regularizer.py", line 66, in __init__
regularizer_blacklist=regularizer_blacklist)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_regularizer_manager.py", line 132, in __init__
['%s (%s)' % (o.name, o.type) for o in self._op_deque])
RuntimeError: OpRegularizerManager could not handle ops: ['base/conv_transpose1/conv2d_transpose (Conv2DBackpropInput)', 'base/conv_output/Sigmoid (Sigmoid)', 'base/conv_output/BiasAdd (BiasAdd)', 'base/conv_output/conv2d_transpose (Conv2DBackpropInput)', 'base/conv_transpose2/Relu (Relu)', 'base/conv_transpose2/BiasAdd (BiasAdd)', 'base/conv_transpose2/conv2d_transpose (Conv2DBackpropInput)', 'base/conv_transpose1/Relu (Relu)', 'base/conv_transpose1/BiasAdd (BiasAdd)', 'base/conv_transpose1/stack (Pack)']
Any thoughts on handling deconvs?
can anyone please tell me how to use the morph-net to find the optimized number of filter size for Tiny-yolo-v3 during training using darknet (https://github.com/AlexeyAB/darknet).
I want to optimize the model for latency (that is my regularizer target cost)
Any documentation or information on how to apply morph-net during darknet training will be highly helpful
I tried morphnet on my Keras-TF implementation of a skip network. Each deep module includes a Keras UpSampling2D
layer, which upsamples a tensor from (batch_size, a, b, channels)
to (batch_size, 2a, 2b, channels)
. That's an issue because the tf ops ResizeNearestNeighbor
and ResizeBilinear
, which are used by that layer, are not supported by morphnet at the moment.
Before trying to implement the op handlers for these I quickly wanted to ask, whether the optimization routine might have inherent issues with upsamplings in general that might not be obvious at first.
Secondly, after reading #36 it is my understanding I could optimize a subnetwork between two upsampling layers by setting the input and output boundaries to these two layers respectively. Is that correct?
Hello,
Thank you for such a wonderful work and open-sourcing it. However, if a working demo code could be added, then naive users like me would find it more comfortable using the library. Therefore, could you please add some working demo code of experiments mentioned in the paper using some seed network.
Hi, I am trying to train a network with the GammaFlopsRegularizer
by distributing it over 4 gpus on my machine. Im using the horovod library to train a simple convolutional autencoder following the script below:
import numpy as np
import os
import tensorflow as tf
import horovod.tensorflow as hvd
from morph_net.network_regularizers import flop_regularizer
from morph_net.network_regularizers import latency_regularizer
from morph_net.tools import structure_exporter
# Define the Network
def conv_ae(x_ph, is_training, scope, channels=[32, 64, 64], reuse=False):
norm_params = {'is_training': is_training, 'scale': True, 'center': False}
# Network Definition
with tf.variable_scope(scope, reuse=reuse):
with slim.arg_scope([slim.conv2d, slim.conv2d_transpose],
normalizer_fn=slim.batch_norm,
normalizer_params=norm_params,
weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
weights_regularizer=slim.l2_regularizer(0.0005)):
conv1 = slim.conv2d(x_ph, num_outputs=channels[0], kernel_size=3, scope='conv1')
pool1 = slim.max_pool2d(conv1, kernel_size=2, scope='pool1')
conv2 = slim.conv2d(pool1, num_outputs=channels[1], kernel_size=3, scope='conv2')
pool2 = slim.max_pool2d(conv2, kernel_size=2, scope='pool2')
conv3 = slim.conv2d(pool2, num_outputs=channels[2], kernel_size=3, scope='conv3')
# deconv part
deconv1 = slim.conv2d_transpose(conv3, num_outputs=channels[2], kernel_size=3, stride=2, scope='conv_transpose1')
deconv2 = slim.conv2d_transpose(deconv1, num_outputs=channels[1], kernel_size=3, stride=2, scope='conv_transpose2')
out = slim.conv2d_transpose(deconv2, num_outputs=1, kernel_size=3, stride=1, normalizer_fn=None, normalizer_params=None, activation_fn=tf.nn.sigmoid, scope='conv_output')
return out
# Arguments
total_steps = 10000
batch_size = 128
val_batch_size = 1000
learning_rate = 1e-3
outdir = 'ae_morph_saves/'
gamma_scale = 1e-3
reg_strength = 1e-8
channels = [64, 64, 64]
# Initialize horovod
hvd.init()
# Load MNIST Data
train_data, test_data = tf.keras.datasets.mnist.load_data()
X_train, X_test = train_data[0], test_data[0]
global_step = tf.train.get_or_create_global_step()
# Optimization
N, H, W = X_train.shape
X_train_ph = tf.placeholder(tf.float32, [None, H, W, 1])
pred = conv_ae(X_train_ph, True, scope='base', channels=channels, reuse=False)
loss = tf.reduce_mean(tf.reduce_sum(tf.squared_difference(X_train_ph, pred), axis=[1,2,3]), axis=0)
model_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES, scope='base')
# applying L1 loss on batch norm gamma
network_regularizer = flop_regularizer.GammaFlopsRegularizer(
output_boundary=[pred.op],
input_boundary=[X_train_ph.op],
gamma_threshold=gamma_scale)
reg_loss = network_regularizer.get_regularization_term() * reg_strength
cost_op = network_regularizer.get_cost()
# setting up distributed optimizer
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate * hvd.size())
optimizer = hvd.DistributedOptimizer(optimizer)
gradients = optimizer.compute_gradients(loss + reg_loss, var_list=model_vars)
train_op = optimizer.apply_gradients(gradients, global_step=global_step)
exporter = structure_exporter.StructureExporter(network_regularizer.op_regularizer_manager)
hooks = [
hvd.BroadcastGlobalVariablesHook(0),
tf.train.StopAtStepHook(last_step=total_steps // hvd.size()),
tf.train.LoggingTensorHook(tensors={'step': global_step, 'loss': loss}, every_n_iter=10),
]
# pin GPU to be used to process local rank (one GPU per process)
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
config.gpu_options.visible_device_list = str(hvd.local_rank())
outdir = outdir if hvd.rank() == 0 else None
# Training Loop
with tf.train.MonitoredTrainingSession(checkpoint_dir=outdir,
hooks=hooks,
config=config) as mon_sess:
while not mon_sess.should_stop():
idx = np.random.choice(X_train.shape[0], batch_size, replace=False)
x_t = np.expand_dims(X_train[idx], axis=-1)
feed_dict = {X_train_ph: x_t}
g_step_val = mon_sess.run(global_step, feed_dict=feed_dict)
_, grad_vals, structure_exporter_tensors, t_loss, reg_cost = mon_sess.run(
[train_op, gradients, exporter.tensors, loss, cost_op],
feed_dict=feed_dict)
print("Step: ", g_step_val)
print("Training Loss: ", t_loss)
print("Reg Cost: ", reg_cost)
# exporting model to JSON
if hvd.rank() == 0:
exporter.populate_tensor_values(structure_exporter_tensors)
exporter.create_file_and_save_alive_counts(outdir, g_step_val)
When trying to train the model using mpirun -np 4 -localhost localhost:4 -bind-to none -map-by slot -genvlist LD_LIBRARY_PATH,PATH python <script-above>
it starts to hang, suggesting that the gradients are out of sync. The error is shown below:
Fri May 31 19:02:24 2019[1,0]<stderr>:[2019-05-31 19:02:24.396204: W horovod/common/operations.cc:588] One or more tensors were submitted to be reduced, gathered or broadcasted by subset of ranks and are waiting for remainder of ranks for more than 60 seconds. This may indicate that different ranks are trying to submit different tensors or that only subset of ranks is submitting tensors, which will cause deadlock.
Fri May 31 19:02:24 2019[1,0]<stderr>:Stalled ops:DistributedAdamOptimizer_Allreduce/HorovodAllreduce_gradients_AddN_3_0 [missing ranks: 0, 1, 3]
Fri May 31 19:02:24 2019[1,0]<stderr>:[2019-05-31 19:02:24.396257: W horovod/common/operations.cc:588] DistributedAdamOptimizer_Allreduce/HorovodAllreduce_gradients_AddN_8_0 [missing ranks: 0, 1, 3]
If I remove the GammaFlopsRegularizer
portion, the error disappears and the model trains fine. My guess is that the regularizer is doing some non-deterministic pruning of connections. There seem to be some nodes where the corresponding gradient has been removed from the graph. I may be wrong, but is there a way to enforce this pruning to be deterministic across the distributed workers.
If not, is there another fix? I think this would be a good thing to look into as most large models that can be pruned rely on some form of distributed training.
In my experimentation when I tried to use Morph-net on google colab there were no changes in the learned-structures while when I tried to run the same code on RTX8000 it was able to optimize the network. Would Morphnet not work on google-colab or is there a restriction to the batch size for morphnet to work.
Hello,
I have tried a few examples from tensorflow/model with morphnet (lenet and resnet), a simple mnist model (https://github.com/mengdong/morph-net/blob/master/morph_net/examples/mnist/mnist-tutorial.py) works. However, I ran into problems in some other more complex models under tensorflow estimator interface.
I wonder is there a recommended way to use morphnet in tf estimator inferface? I know there is quite some overhead in the estimator's graph. Detailed infromation below:
Regarding lenet (https://github.com/mengdong/morph-net/blob/master/morph_net/examples/mnist/mnist.py) from https://github.com/tensorflow/models/tree/master/official/mnist, I observe that:
I0904 13:14:27.240477 140031449261888 op_regularizer_manager.py:125]
OpRegularizerManager found 63 ops and 4 sources.
......
File "/home/dongm/workspace/laptop_mapping/morph-net/morph_net/framework/op_regularizer_manager.py", line 137, in __init__
['%s (%s)' % (o.name, o.type) for o in self._op_deque])
RuntimeError: OpRegularizerManager could not handle ops: ['sequential/conv2d/BiasAdd (BiasAdd)', 'sequential/max_pooling2d_1/MaxPool (MaxPool)', 'sequential/conv2d_1/BiasAdd (BiasAdd)', 'sequential/max_pooling2d/MaxPool (MaxPool)', 'sequential/conv2d/BiasAdd/ReadVariableOp (ReadVariableOp)']
Regarding ResNet (https://github.com/mengdong/morph-net/blob/master/morph_net/examples/resnet/imagenet_main.py), I observe:
I0904 11:27:34.397989 139699288442688 op_regularizer_manager.py:125]
OpRegularizerManager found 629 ops and 53 sources.
.....
RuntimeError: OpRegularizerManager could not handle ops:
['resnet_model/batch_normalization_45/FusedBatchNormV3 (FusedBatchNormV3)',
'resnet_model/Pad_6 (Pad)', 'resnet_model/batch_normalization_44/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_49/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_48/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_47/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_52/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_51/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/Squeeze (Squeeze)', 'resnet_model/final_reduce_mean (Identity)', 'resnet_model/Mean (Mean)', 'resnet_model/batch_normalization_50/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_43/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_24/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_11/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_1/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/Pad (Pad)',
'resnet_model/batch_normalization/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_4/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_3/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/max_pooling2d/MaxPool (MaxPool)', 'resnet_model/initial_max_pool (Identity)', 'resnet_model/batch_normalization_2/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_7/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_6/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_5/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_10/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_9/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_8/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_14/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_13/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/Pad_2 (Pad)', 'resnet_model/batch_normalization_12/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_17/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_16/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_15/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_20/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_19/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_18/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_23/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_22/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_21/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_27/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_26/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/Pad_4 (Pad)',
'resnet_model/batch_normalization_25/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_30/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_29/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_28/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_33/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_32/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_31/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_36/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_35/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_34/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_39/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_38/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_37/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_42/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_41/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_40/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_46/FusedBatchNormV3 (FusedBatchNormV3)', 'resnet_model/batch_normalization_45/ReadVariableOp (ReadVariableOp)', 'resnet_model/batch_normalization_45/ReadVariableOp_1 (ReadVariableOp)']
Hi
Thanks a lot for making morphnet open source. I wish to try it out but it seems I am having some trouble with getting it to work.
I have multiple outputs from my network. I am using the GroupLassoFlopsRegularizer and it seems to give me an error when i pass it a list of logits which come out of my network.
My network
`
def build_model(self):
self.g = tf.Graph()
with self.g.as_default():
tf.set_random_seed(42)
self.inputs = tf.placeholder(tf.float32, [None, *(self.state_size)], name="inputs_")
print(self.inputs)
self.out_a = tf.placeholder(tf.int32, [None, ], name="out_a")
self.out_b = tf.placeholder(tf.int32, [None, ], name="out_b")
self.out_c = tf.placeholder(tf.int32, [None, ], name="out_c")
self.out_d = tf.placeholder(tf.int32, [None, ], name="out_d")
self.discounted_episode_rewards_norm = tf.placeholder(tf.float32, [None, ],
name="discounted_episode_rewards_")
self.avg_reward_ = tf.placeholder(tf.float32, name="avg_reward")
print(self.discounted_episode_rewards_norm)
# Add this placeholder for having this variable in tensorboard
self.mean_reward_ = tf.placeholder(tf.float32, name="mean_reward")
#model=k.Sequential()
#inp = Input(shape=self.state_size, name='stack_of_4_images')(self.inputs)
c1 = Conv2D(32,8,strides=2,activation='relu',name='Conv1')(self.inputs) #kernel_initializer=glorot_uniform(seed=42),
c2 = Conv2D(64, (6,8), strides=2, activation='relu', name='Conv2')(c1) # kernel_initializer=glorot_uniform(seed=42),
c3 = Conv2D(128, (6,8), strides=2, activation='relu', name='Conv3')(c2) # kernel_initializer=glorot_uniform(seed=42),
flat = Flatten()(c3)
d1 = Dense(1024, activation='selu', name='Dense1')(flat)
d2 = Dense(1024, activation='selu', name='Dense2')(d1)
logits_a = Dense(self.output_size, name='logits_a')(d2)
logits_b = Dense(self.output_size, name='logits_b')(d2)
logits_c = Dense(self.output_size, name='logits_c')(d2)
logits_d = Dense(self.output_size, name='logits_d')(d2)
self.outputs_softmax_a= Softmax()(logits_a)
self.outputs_softmax_b = Softmax()(logits_b)
self.outputs_softmax_c = Softmax()(logits_c)
self.outputs_softmax_d = Softmax()(logits_d)
self.network_regularizer = flop_regularizer.GroupLassoFlopsRegularizer(
[logits_a.op, logits_b.op, logits_c.op, logits_d.op], threshold=1e-3)
self.exporter = structure_exporter.StructureExporter(
self.network_regularizer.op_regularizer_manager)
self.regularization_strength = 1e-10
with tf.name_scope("loss"):
neg_log_prob_a = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_a, labels=self.out_a)
neg_log_prob_b = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_b, labels=self.out_b)
neg_log_prob_c = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_c, labels=self.out_c)
neg_log_prob_d = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_d, labels=self.out_d)
loss_a = tf.reduce_mean(neg_log_prob_a * self.discounted_episode_rewards_norm) # reward guided loss
loss_b = tf.reduce_mean(neg_log_prob_b * self.discounted_episode_rewards_norm) # reward guided loss
loss_c = tf.reduce_mean(neg_log_prob_c * self.discounted_episode_rewards_norm) # reward guided loss
loss_d = tf.reduce_mean(neg_log_prob_d * self.discounted_episode_rewards_norm) # reward guided loss
self.regularizer_loss = (self.network_regularizer.get_regularization_term() *
self.regularization_strength)
self.loss = loss_a + loss_b + loss_c + loss_d + self.regularizer_loss
with tf.name_scope("train"):
#update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
train_ops = tf.train.AdamOptimizer(self.lr).minimize(self.loss)
self.train_op = tf.group([train_ops])#, update_ops])
self.setup_writer() # has to be before global variables initializer inside the graph for write_op to be
# populated
self.init = tf.global_variables_initializer()`
The error i get when i run my script that instantiates the model
File "/home/ubuntu/morph_basic/basic_bline.py", line 78, in build_model [logits_a.op, logits_b.op, logits_c.op, logits_d.op], threshold=1e-3) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/morph_net/network_regularizers/flop_regularizer.py", line 153, in __init__ regularizer_blacklist=regularizer_blacklist) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/morph_net/framework/op_regularizer_manager.py", line 126, in __init__ self._op_handler_dict[op.type].assign_grouping(op, self) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/morph_net/framework/grouping_op_handler.py", line 40, in assign_grouping output_ops, op_reg_manager) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/morph_net/framework/op_handler_util.py", line 73, in get_ops_without_groups op_slices = op_reg_manager.get_op_slices(op) File "/home/ubuntu/anaconda3/envs/tensorflow_p36/lib/python3.6/site-packages/morph_net/framework/op_regularizer_manager.py", line 438, in get_op_slices if size > 0: TypeError: '>' not supported between instances of 'NoneType' and 'int'
The setup.py
in the repo says the code is under the MIT license:
"License :: OSI Approved :: MIT License",
while the LICENSE
file contains the Apache 2.0 license, which is confusing.
Could you please clarify which license this repo is under and fix one or the other file to ensure consistency?
Thanks!
Hi,
I am currently trying to shrink a segmentation model which uses grouped convolutions with the GammaFlopsRegularizer
. The grouped convolutions are implemented as follows:
+-----------+
| op: split |
+-----------+
|
+----------------------+
| |
+------------+ +------------+
| op: conv2d | ... | op: conv2d |
+------------+ +------------+
| |
+----------------------+
|
+------------+
| op: concat |
+------------+
|
+---------------+
| op: batchnorm |
+---------------+
Using morph-net on a model which contains such layers/ structures causes an endless loop during regularizer generation. The loop seems to be caused by the split operation which expands the group rather than closing it (which makes totally sense usually).
My current workaround is adding the Split operation as a OutputNonPassthroughOpHandler
in the _get_base_op_handler_dicts()
function of the framework/op_handlers.py
module. This seems to stop the endless loop but I am not sure it this produces the correct regularizer groups.
As this "workaround" seems rather hacky and potentially breaks the grouping for other structures, I am curious if you see a better way to solve this problem?
Thank you for your help and your awesome framework!
when I use multiple output like this
network_regularizer = flop_regularizer.GammaFlopsRegularizer([box_pred.op, cls_pred.op], gamma_threshold=1e-3)
I got this error
INFO:tensorflow:OpRegularizerManager starting analysis from: [<tf.Operation 'concat_2' type=ConcatV2>, <tf.Operation 'concat_3' type=ConcatV2>].
INFO:tensorflow:OpRegularizerManager found 3414 ops and 61 sources.
Total size for op slices do not match: %s' % op_slice_sizes)
ValueError: Total size for op slices do not match: [[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3], [3]]
I am confused so much. Thank you for reply!
the shape of outputs are:
<class 'tensorflow.python.framework.ops.Tensor'> (48, 2046, 4)
<class 'tensorflow.python.framework.ops.Tensor'> (48, 2046, 4)
@batzner @eladeban
Would it be possible to make this package easily accessible through pip.
And is there any example released?
hi, I have a question about the expansion.
I trained the regularizer_loss and got the following network structure.
How should I use a uniform width multiplier to do the expansion?
Do I need to manually adjust the number of channels, if I use the multiplier of 2, does that mean I need to set the number of the layer('g_net/enc1_2/conv1/Conv2D: 12')'s channels to 24.
Also, I would like to ask if morph-net can be used for pixel to pixel tasks, such as super resolution and de-blur. Can keep the same accuracy after shrinking the network?
As the proposed structure had been learned and exported in JSON, meanwhile the weights had been also optimized. Is there any way to save and restore the trained weights along with proposed structure, instead of retrain for scratch? I think it is very helpful and time-saving.
Am getting all the learned structures same as my initial network , i.e morphnet did not change my network at all. what could be the possible reasons for this.. and how to correct it ?
thank you for your excellent work.
Here I decide to realize this algorithm by myself according to this paper in MobileNetV2 ,and met one problems.
In the case of using model_size regularizer , C_cost of every layer should be computed to define the cost function.And we train the model to decide how to shrink the Network.
But I notice that in MobileNetV2 , the input and output channels should be kept same in depthwise conv-layer. Is this means I should ignore this layer when shrink the model?
and in this case when computing the lambda in cost function, I should also ignore this part's model size.
thank you very much.
Hi, thanks for sharing open source.
Actually I got a problem with FLOPs when I'm training resnet_cifar_10 with PocketFlow.
I try several combinations of regularization_strength and gamma_threshold.
want to ask if this is correct?
Code for my above implementation is provided below :
`
# data input pipeline
with tf.variable_scope(self.data_scope):
iterator = self.build_dataset_train() if is_train else self.build_dataset_eval()
images, labels = iterator.get_next()
if not isinstance(images, dict):
tf.add_to_collection('images_final', images)
else:
tf.add_to_collection('images_final', images['image'])
# model definition - distilled model
if self.enbl_dst:
logits_dst = self.helper_dst.calc_logits(sess, images)
# model definition - primary model
with tf.variable_scope(self.model_scope):
# forward pass
if is_train and self.forward_w_labels:
logits = self.forward_train(images, labels)
else:
logits = self.forward_train(images) if is_train else self.forward_eval(images)
if not isinstance(logits, dict):
tf.add_to_collection('logits_final', logits)
else:
for value in logits.values():
tf.add_to_collection('logits_final', value)
# loss & extra evalution metrics
loss, metrics = self.calc_loss(labels, logits, self.trainable_vars)
if self.enbl_dst:
loss += self.helper_dst.calc_loss(logits, logits_dst)
tf.summary.scalar('loss', loss)
for key, value in metrics.items():
tf.summary.scalar(key, value)
# optimizer & gradients
if is_train:
self.global_step = tf.train.get_or_create_global_step()
lrn_rate, self.nb_iters_train = self.setup_lrn_rate(self.global_step)
network_regularizer = flop_regularizer.GammaFlopsRegularizer(
output_boundary = [logits.op],
input_boundary=[images.op, labels.op],
gamma_threshold=8.8e-1 #
)#
exporter = structure_exporter.StructureExporter(network_regularizer.op_regularizer_manager)
regularization_strength = 1e-8 #
regularizer_loss = (network_regularizer.get_regularization_term() * regularization_strength)#
model_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(logits = logits, labels= tf.argmax(tf.cast(labels, dtype=tf.int32),1))#
optimizer = tf.train.MomentumOptimizer(lrn_rate, FLAGS.momentum)
train_op = optimizer.minimize(model_loss + regularizer_loss)#
tf.summary.scalar('RegularizationLoss', regularizer_loss)#
tf.summary.scalar(network_regularizer.cost_name, network_regularizer.get_cost())
tf.summary.FileWriter(FLAGS.log_dir, graph=tf.get_default_graph())
exporter = structure_exporter.StructureExporter(network_regularizer.op_regularizer_manager)
sess.run(tf.global_variables_initializer())
for step in range(FLAGS.steps):
_, structure_exporter_tensors = sess.run([train_op, exporter.tensors])
if (step % 1000 == 0):
exporter.populate_tensor_values(structure_exporter_tensors)
exporter.create_file_and_save_alive_counts(FLAGS.log_dir, step)
`
This model build by keras. and i use keras.layers.BatchNormaliztion.
But when i type this code, i get this error
with tf.name_scope('Morph_net'):
network_regularizer = flop_regularizer.GammaFlopsRegularizer(
output_boundary=[out.op],
input_boundary=[left_input.op, right_input.op],
gamma_threshold= 0.45,
regularizer_blacklist= None,
regularizer_decorator=dummy_decorator.DummyDecorator,
decorator_parameters={'scale': 0.5}
)
regularization_strength = 1e-10
regularizer_loss = (network_regularizer.get_regularization_term() * regularization_strength)
All error massage is like this
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-69-27398095666a> in <module>
161 regularizer_blacklist= None,
162 regularizer_decorator=dummy_decorator.DummyDecorator,
--> 163 decorator_parameters={'scale': 0.5}
164 )
165 regularization_strength = 1e-10
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/network_regularizers/flop_regularizer.py in __init__(self, output_boundary, gamma_threshold, regularizer_decorator, decorator_parameters, input_boundary, force_group, regularizer_blacklist)
70 input_boundary=input_boundary,
71 force_group=force_group,
---> 72 regularizer_blacklist=regularizer_blacklist)
73 self._calculator = cost_calculator.CostCalculator(
74 self._manager, resource_function.flop_function)
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/framework/op_regularizer_manager.py in __init__(self, output_boundary, op_handler_dict, create_grouping_regularizer, force_group, regularizer_blacklist, input_boundary, iteration_limit)
168 handler = self._op_handler_dict[source_op_slice.op.type]
169 source_op_slices.append(source_op_slice)
--> 170 regularizers.append(handler.create_regularizer(source_op_slice))
171
172 # Create a group regularizer and assign to all OpSlice in the OpGroup. If
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/framework/op_handler_decorator.py in create_regularizer(self, op_slice)
51 A decorated OpRegularizer for the given OpSlice.
52 """
---> 53 regularizer = self._op_handler.create_regularizer(op_slice)
54 if regularizer and self._regularization_decorator:
55 regularizer = self._regularization_decorator(
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/framework/batch_norm_source_op_handler.py in create_regularizer(self, op_slice)
62 if start_index == 0 and size == gamma.shape.as_list()[-1]:
63 return gamma_l1_regularizer.GammaL1Regularizer(
---> 64 gamma, self._gamma_threshold)
65 else:
66 # Note: this conversion is also attempted inside GammaL1Regularizer
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/op_regularizers/gamma_l1_regularizer.py in __init__(self, gamma, gamma_threshold)
21 considered 'alive'.
22 """
---> 23 self._gamma = tpu_util.maybe_convert_to_variable(gamma)
24 self._gamma_threshold = gamma_threshold
25 abs_gamma = tf.abs(self._gamma)
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/framework/tpu_util.py in maybe_convert_to_variable(tensor)
74 reuse=True,
75 ):
---> 76 variable_name = get_variable_name(op)
77 tf.logging.info('Converting tensor %s --> variable %s',
78 tensor, variable_name)
~/anaconda3/lib/python3.7/site-packages/morph_net-0.2.1-py3.7.egg/morph_net/framework/tpu_util.py in get_variable_name(read_variable_op)
37 op = read_variable_op
38 while op.type != 'VarHandleOp':
---> 39 assert len(op.inputs) == 1
40 op = op.inputs[0].op
41 return op.name
AssertionError:
Would you answer me how can i fix this?
Just like the title,I have a model trained by Pytorch, Can i use this project to slim it..
Eg: Convert Pytorch's model to ONNX, then read by Tensorflow?
Thanks!
Hi, I got a problem when I used the VGG. The code is as follows:
def vgg_a(inputs,num_classes=10,is_training=True,dropout_keep_prob=0.5,spatial_squeeze=True,
scope='vgg_a',fc_conv_padding='SAME',global_pool=False):
.........
net = slim.conv2d(net,512,[3,3],scope='conv5_2')
net = slim.max_pool2d(net,[2,2],scope='pool5')
net = slim.conv2d(net,4096,[1,1],padding=fc_conv_padding,scope='fc6')
net = slim.dropout(net,dropout_keep_prob,is_training=is_training,scope='dropout6')
net = slim.conv2d(net,4096,[1,1],scope='fc7')
if num_classes:
net = slim.dropout(net,dropout_keep_prob,is_training=is_training,scope='dropout7')
net = slim.conv2d(net,num_classes,[1,1],activation_fn=None,normalizer_fn=None,scope='fc8')
return net
if __name__ == '__main__':
train_x,train_y,test_x,test_y = prepare_data()
x = tf.placeholder(tf.float32,[None,image_size,image_size,3])
y_ = tf.placeholder(tf.float32,[None,class_num])
dropout_keep_prob = tf.placeholder(tf.float32)
learning_rate = tf.placeholder(tf.float32)
is_training = tf.placeholder(tf.bool)
logits,_ = vgg_a(inputs=x, dropout_keep_prob=dropout_keep_prob)
network_regularizer = flop_regularizer.GroupLassoFlopsRegularizer([logits.op], threshold=1e-3)
regularization_strength = 1e-7
regularizer_loss = (network_regularizer.get_regularization_term() * regularization_strength)
.........
_,batch_loss = sess.run([train_step,cross_entropy],feed_dict={
x:batch_x, y_:batch_y, dropout_keep_prob:dropout_rate, learning_rate:lr,
is_training: True})
batch_acc = accuracy.eval(feed_dict={x:batch_x, y_:batch_y, dropout_keep_prob: 1.0,
is_training: True})
I would like to ask what is the cause of this error, does not support dropout? “logits,_ = vgg_a(inputs=x)” will not be wrong, but dropout requires different parameters in the train and accuracy eval. Is there any solution?
Thanks!
Generally, we use a placeholder for the is_training
flag of normalizer_params
for batch norm to toggle between training and test mode. When using morph net regularization though, we can't set is_training
to a placeholder since we see the following error:
Traceback (most recent call last):
File "gamma_prune.py", line 47, in <module>
network_regularizer = latency_regularizer.GammaLatencyRegularizer([out.op], gamma_threshold=gamma_scale, hardware='K80')
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/network_regularizers/latency_regularizer.py", line 90, in __init__
force_group=force_group, regularizer_blacklist=regularizer_blacklist)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_regularizer_manager.py", line 126, in __init__
self._op_handler_dict[op.type].assign_grouping(op, self)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/grouping_op_handler.py", line 35, in assign_grouping
input_ops, op_reg_manager)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_handler_util.py", line 73, in get_ops_without_groups
op_slices = op_reg_manager.get_op_slices(op)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_regularizer_manager.py", line 437, in get_op_slices
size = op_handler_util.get_op_size(op)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_handler_util.py", line 505, in get_op_size
shape = op.outputs[0].shape.as_list()
File "/Users/vashishtmadhavan/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/tensor_shape.py", line 904, in as_list
raise ValueError("as_list() is not defined on an unknown TensorShape.")
ValueError: as_list() is not defined on an unknown TensorShape.
What's the appropriate strategy here? Would we just set is_training
to False for both training and testing or True?
Code:
`import tensorflow as tf
from morph_net.network_regularizers import flop_regularizer
from morph_net.network_regularizers import latency_regularizer
from morph_net.tools import structure_exporter
import numpy as np
from datetime import datetime
import time
def conv2d(input, w, b):
return tf.nn.relu(tf.nn.bias_add(tf.nn.conv2d(input, w, strides=[1, 1, 1, 1], padding='SAME'),
b))
def max_pool(input):
return tf.nn.max_pool(input, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1],
padding='SAME')
def fc(input, w, b):
return tf.nn.relu(tf.add(tf.matmul(input, w), b))
def alex_net(_input_r, _weights, _biases, _keep_prob):
with tf.name_scope('conv1'):
_conv1 = conv2d(_input_r, _weights['wc1'], _biases['bc1'])
with tf.name_scope('pool1'):
_pool1 = max_pool(_conv1)
with tf.name_scope('conv2'):
_conv2 = conv2d(_pool1, _weights['wc2'], _biases['bc2'])
with tf.name_scope('pool2'):
_pool2 = max_pool(_conv2)
with tf.name_scope('conv3'):
_conv3 = conv2d(_pool2, _weights['wc3'], _biases['bc3'])
with tf.name_scope('conv4'):
_conv4 = conv2d(_conv3, _weights['wc4'], _biases['bc4'])
with tf.name_scope('conv5'):
_conv5 = conv2d(_conv4, _weights['wc5'], _biases['bc5'])
with tf.name_scope('pool3'):
_pool3 = max_pool(_conv5)
_densel = tf.reshape(_pool3,
[-1, _weights['wd1'].get_shape().as_list()[0]])
with tf.name_scope('fc1'):
_fc1 = fc(_densel, _weights['wd1'], _biases['bd1'])
_fc1_drop = tf.nn.dropout(_fc1, _keep_prob)
with tf.name_scope('fc2'):
_fc2 = fc(_fc1_drop, _weights['wd2'], _biases['bd2'])
_fc2_drop = tf.nn.dropout(_fc2, _keep_prob)
with tf.name_scope('out'):
_out = tf.add(tf.matmul(_fc2_drop, _weights['wd3']), _biases['bd3'])
return _out
n_output = 10
learning_rate = 0.001
dropout = 0.75
train_data, test_data = tf.keras.datasets.mnist.load_data()
X_train, X_test = train_data[0], test_data[0]
Y_train, Y_test = train_data[1], test_data[1]
train_dir = "./Morph_save/"
print("CNN READY")
x = tf.placeholder(tf.float32, [None, 28, 28, 1])
y = tf.placeholder(tf.float32, [None, 10])
keep_prob = tf.placeholder(tf.float32)
weights = {
'wc1': tf.Variable(tf.truncated_normal([3, 3, 1, 64], dtype=tf.float32, stddev=0.1), name='weights1'),
'wc2': tf.Variable(tf.truncated_normal([3, 3, 64, 128], dtype=tf.float32, stddev=0.1), name='weights2'),
'wc3': tf.Variable(tf.truncated_normal([3, 3, 128, 256], dtype=tf.float32, stddev=0.1), name='weights3'),
'wc4': tf.Variable(tf.truncated_normal([3, 3, 256, 256], dtype=tf.float32, stddev=0.1), name='weights4'),
'wc5': tf.Variable(tf.truncated_normal([3, 3, 256, 128], dtype=tf.float32, stddev=0.1), name='weights5'),
'wd1': tf.Variable(tf.truncated_normal([4*4*128, 1024], dtype=tf.float32, stddev=0.1), name='weights_fc1'),
'wd2': tf.Variable(tf.random_normal([1024, 1024], dtype=tf.float32, stddev=0.1), name='weights_fc2'),
'wd3': tf.Variable(tf.random_normal([1024, n_output], dtype=tf.float32, stddev=0.1), name='weights_output')
}
biases = {
'bc1': tf.Variable(tf.constant(0.0, shape=[64], dtype=tf.float32), trainable=True, name='biases1'),
'bc2': tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32), trainable=True, name='biases2'),
'bc3': tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases3'),
'bc4': tf.Variable(tf.constant(0.0, shape=[256], dtype=tf.float32), trainable=True, name='biases4'),
'bc5': tf.Variable(tf.constant(0.0, shape=[128], dtype=tf.float32), trainable=True, name='biases5'),
'bd1': tf.Variable(tf.constant(0.0, shape=[1024], dtype=tf.float32), trainable=True, name='biases_fc1'),
'bd2': tf.Variable(tf.constant(0.0, shape=[1024], dtype=tf.float32), trainable=True, name='biases_fc2'),
'bd3': tf.Variable(tf.constant(0.0, shape=[n_output], dtype=tf.float32), trainable=True, name='biases_output')
}
print("CNN READY")
pred = alex_net(x, weights, biases, keep_prob)
network_regularizer = flop_regularizer.GroupLassoFlopsRegularizer(output_boundary=[pred.op],input_boundary=[x.op, y.op],threshold=1e-3)
regularization_strength = 1e-8
regularizer_loss = (network_regularizer.get_regularization_term() * regularization_strength)
model_loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(y, pred))
cost_op = network_regularizer.get_cost()
optimizer = tf.train.MomentumOptimizer(learning_rate=0.01, momentum=0.9)
train_op = optimizer.minimize(model_loss + regularizer_loss)
exporter = structure_exporter.StructureExporter(network_regularizer.op_regularizer_manager)
init = tf.global_variables_initializer()
print("FUNCTIONS READY")
training_epochs = 1000
batch_size = 500
display_step = 10
sess = tf.Session()
sess.run(init)
for epoch in range(training_epochs):
avg_cost = 0.
total_batch = int(60000/batch_size)
start_time = time.time()
for i in range(total_batch):
batch_xs, batch_ys = X_train[ibatch_size:ibatch_size+batch_size], Y_train[ibatch_size:ibatch_size+batch_size]
_, get_cost, loss_val, regloss_val, structure_exporter_tensors=\
sess.run([train_op,cost_op,model_loss,regularizer_loss,exporter.tensors], feed_dict={x: batch_xs, y: batch_ys, keep_prob:dropout})
if epoch % display_step == 0:
train_accuracy = sess.run(model_loss, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 1.0})
test_accuracy = sess.run(model_loss, feed_dict={x: X_train, y: Y_test, keep_prob:1.0})
exporter.populate_tensor_values(structure_exporter_tensors)
exporter.create_file_and_save_alive_counts(train_dir, epoch)
print("Epoch: %03d/%03d cost: %.9f TRAIN ACCURACY: %.3f TEST ACCURACY: %.3f" % (epoch, training_epochs, avg_cost, train_accuracy, test_accuracy))
duration = time.time() - start_time
print('%s: step %d, duration = %.3f' % (datetime.now(), epoch, duration))
print("DONE")
`
Hi, I've been trying to train a simple model on MNIST to test the out the GammaLatencyRegularizer
. Since I'm testing this on CPU I'm passing FLOP_LATENCY
as the hardware
argument. However, it seems to give me the following error:
Traceback (most recent call last):
File "gamma_prune.py", line 43, in <module>
network_regularizer = latency_regularizer.GammaLatencyRegularizer([out.op], gamma_threshold=1e-3, hardware='FLOP_LATENCY')
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/network_regularizers/latency_regularizer.py", line 92, in __init__
force_group=force_group, regularizer_blacklist=regularizer_blacklist)
File "/Users/vashishtmadhavan/Documents/Code/morph-net/morph_net/framework/op_regularizer_manager.py", line 132, in __init__
['%s (%s)' % (o.name, o.type) for o in self._op_deque])
RuntimeError: OpRegularizerManager could not handle ops: ['base/Flatten/flatten/Reshape (Reshape)']
However, when I train the same model using GammaFlopsRegularizer
it seems to work fine. Any comment on this issue?
Also, here is my network def'n
def base_model(x_ph, is_training_ph, scope, channels=[32, 64, 64], reuse=False):
norm_params = {'is_training': False, 'scale': True, 'center': False}
# Network Definition
with tf.variable_scope(scope, reuse=reuse):
with slim.arg_scope([slim.conv2d, slim.fully_connected],
normalizer_fn=slim.batch_norm,
normalizer_params=norm_params,
weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
weights_regularizer=slim.l2_regularizer(0.0005)):
conv1 = slim.conv2d(x_ph, num_outputs=channels[0], kernel_size=3, scope='conv1')
pool1 = slim.max_pool2d(conv1, kernel_size=2, scope='pool1')
conv2 = slim.conv2d(pool1, num_outputs=channels[1], kernel_size=3, scope='conv2')
pool2 = slim.max_pool2d(conv2, kernel_size=2, scope='pool2')
conv3 = slim.conv2d(pool2, num_outputs=channels[2], kernel_size=3, scope='conv3')
conv3_flat = slim.flatten(conv3)
out = slim.fully_connected(conv3_flat, num_outputs=10, normalizer_fn=None, normalizer_params=None,
activation_fn=None, scope='output')
pred = tf.argmax(out, axis=1)
return out, pred
Hi,thank you very much for open this project.
Here is my error.This is my model:
from keras.models import Model
from keras.layers import Dense, GlobalAveragePooling2D, Dropout
self.base_model = keras.applications.inception_v3.InceptionV3(include_top=False,weights='inception_v3_weights_tf_dim_ordering_tf_kernels_notop.h5',input_shape=input_shape)
x = self.base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(self.config.fc_size, activation='relu',)(x)
features = Dense(self.config.fc_size, activation='relu',name='features')(Dropout(self.config.dropout_ratio)(x))
predictions = Dense(1, activation='sigmoid',name='predictions')(features)
self.model = Model(input=self.base_model.input, output=[predictions,features])
network_regularizer = flop_regularizer.GammaFlopsRegularizer(
[self.model.output[0].op], gamma_threshold=1e-3)
When I run this ,it returns an error:
File "E:\20190425_MorphNet\20190425_morph-net-master\model_local.py", line 59, in add_new_last_layer
flop_regularizer.GammaFlopsRegularizer([self.model.output[0].op], gamma_threshold=1e-3)
File "E:\20190425_MorphNet\20190425_morph-net-master\morph_net\network_regularizers\flop_regularizer.py", line 95, in __init__
regularizer_blacklist=regularizer_blacklist)
File "E:\20190425_MorphNet\20190425_morph-net-master\morph_net\framework\op_regularizer_manager.py", line 132, in __init__
['%s (%s)' % (o.name, o.type) for o in self._op_deque])
RuntimeError: OpRegularizerManager could not handle ops: ['dropout_1/cond/dropout/Shape (Shape)', 'dropout_2/cond/dropout/Shape (Shape)']
How could I solve this error?
And I have another question, can I use MorphNet on Keras model?
AttributeError Traceback (most recent call last)
in ()
4 logits = build_model()
5
----> 6 network_regularizer = flop_regularizer.GammaFlopsRegularizer( output_boundary=[logits.op],
7 gamma_threshold=1e-3)
8
AttributeError: 'Sequential' object has no attribute 'op'
Hi there, First of all excellent work. I am curious if there are any examples available on using it for TF.Keras model which uses transfer learning (fine-tuned) with pre-trained model such as resnet or inceptionv3? and what is the current support for tensorflow 2.0. I saw there it still uses sessions in the example code section.
Hi, thanks for open souring.
I added these two lines of code,“tf.summary.scalar('RegularizationLoss', regularizer_loss)
tf.summary.scalar(network_regularizer.cost_name, network_regularizer.get_cost())”
but I cannot observe them in tensorboard.
Does it still require some other operations?
Thanks for the answer.
In this case, MorphNet is getting stuck because it doesn't know how to handle the Shape op. You can try a couple options:
Originally posted by @ayp-google in #36 (comment)
I wonder if we can treat the number of ops for the problematic Shape op as 0?
I have a Keras Sequential Model and I train the model using,
model.fit(x_train.values, y_train.values, batch_size=1024, epochs=10, verbose=2, validation_data=(x_valid,y_valid))
I have configured morphnet to this model. In this case, is there a way to save the model structure using structure exporter?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.