Hello thanks for this great respository! I use the GammaFlopsReg

Regularizer Loss remains zero / BatchToSpaceND about morph-net HOT 1 OPEN

gargahcerk commented on August 24, 2024

Regularizer Loss remains zero / BatchToSpaceND

from morph-net.

Comments (1)

gargahcerk commented on August 24, 2024

I thought the problem was with BatchToSpace. I was able to prevent BatchToSpace from occurring (occurs when conv1d dilation > 1) and there is still the problem with Regularizer Loss = 0.0. The FLOPs can be calculated, so I assume that reg_outputs/ reg_inputs (network_regularizers/resource_function.py#L477) must be 0.

Further information:
I use the model Jasper (Openseq2seq). For Batch Normalization I set scale and fused = true. channels_last is set.

This is how I included morphnet:

def get_regularization_loss(self):
    from open_seq2seq.optimizers import get_regularization_loss as old_regularization_loss
    if not self.params['morphnet']:
      return old_regularization_loss()
    else:
      from morph_net.network_regularizers import flop_regularizer
      from morph_net.tools import structure_exporter

      network_regularizer = flop_regularizer.GammaFlopsRegularizer(output_boundary=[self._logits.op], input_boundary=[self.get_data_layer().input_tensors["source_tensors"][0].op,  self.get_data_layer().input_tensors["target_tensors"][0].op], gamma_threshold=1e-3)

      regularization_strength = 1e-10
      regularizer_loss = (network_regularizer.get_regularization_term() * regularization_strength)
      self.reg_loss = regularizer_loss
      self.reg_costs = network_regularizer.get_cost()

      tf.summary.scalar('RegularizationLoss', regularizer_loss)
      tf.summary.scalar(network_regularizer.cost_name, network_regularizer.get_cost())

      return tf.add(old_regularization_loss(),regularizer_loss)

from morph-net.

Recommend Projects