抱歉又来打扰您。训练多个任务,freeze_step设置为>0时,会报一个如题所述的异常;自己尝试调试,但最终未能解决,希望能得到你的建议。
环境:
- python 3.6
- tensorflow-gpu 1.15.0 (pip install bert-multitask-learning 之后,运行示例,提示需要tensorflow版本>=1.14,故升级为1.15了)
何时遇到 ValueError:
- 训练单个problem,freeze_step设置为0 或者>0,均没有报异常
- 训练两个problem ('cls|seq_tag'),freeze_step设置为0没有报异常,但是设置为>0,就报ValueError异常
- 异常显示是IndexedSlicesSpec不匹配,具体Traceback如下
Traceback (most recent call last):
File "debug_freeze.py", line 161, in
model_dir=model_dir)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/bert_multitask_learning/run_bert_multitask.py", line 126, in train_bert_multitask
train_and_evaluate(estimator, train_spec, eval_spec)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 473, in train_and_evaluate
return executor.run()
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 613, in run
return self.run_local()
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/training.py", line 714, in run_local
saving_listeners=saving_listeners)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1159, in _train_model
return self._train_model_distributed(input_fn, hooks, saving_listeners)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1222, in _train_model_distributed
self._config._train_distribute, input_fn, hooks, saving_listeners)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1302, in _actual_train_model_distributed
self.config))
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1810, in call_for_each_replica
return self._call_for_each_replica(fn, args, kwargs)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 662, in _call_for_each_replica
fn, args, kwargs)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 196, in _call_for_each_replica
coord.join(threads)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/six.py", line 703, in reraise
raise value
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/training/coordinator.py", line 297, in stop_on_exception
yield
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/distribute/mirrored_strategy.py", line 880, in run
self.main_result = self.main_fn(*self.main_args, **self.main_kwargs)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/bert_multitask_learning/model_fn.py", line 496, in model_fn
features, hidden_feature, loss_eval_pred, mode, warm_start)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/bert_multitask_learning/model_fn.py", line 457, in create_spec
train_scaffold)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/bert_multitask_learning/model_fn.py", line 384, in create_train_spec
aggregation_method=tf.AggregationMethod.EXPERIMENTAL_TREE)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
unconnected_gradients)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 350, in _MaybeCompile
return grad_fn() # Exit early
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in
lambda: grad_fn(op, *out_grads))
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_grad.py", line 84, in _SwitchGrad
return merge(grad, name="cond_grad")[0], None
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/ops/control_flow_ops.py", line 413, in merge
nest.assert_same_structure(inputs[0], v, expand_composites=True)
File "/home/jp/anaconda3/envs/jp_test/lib/python3.6/site-packages/tensorflow_core/python/util/nest.py", line 326, in assert_same_structure
% (str(e), str1, str2))
ValueError: The two structures don't have the same nested structure.
First structure: type=IndexedSlices str=IndexedSlices(indices=Tensor("gradients/cond/Merge_grad/cond_grad/Switch_1:0", shape=(?,), dtype=int64, device=/replica:0/task:0/device:GPU:0), values=Tensor("gradients/cond/Merge_grad/cond_grad/Switch:0", shape=(?, ?, 768), dtype=float32, device=/replica:0/task:0/device:GPU:0), dense_shape=Tensor("gradients/cond/Merge_grad/cond_grad/Switch_2:0", shape=(3,), dtype=int64, device=/replica:0/task:0/device:GPU:0))
Second structure: type=IndexedSlices str=IndexedSlices(indices=Tensor("gradients/cond/Switch_1_grad/cond_grad/Cast:0", shape=(?,), dtype=int64, device=/replica:0/task:0/device:GPU:0), values=Tensor("gradients/zeros:0", shape=(?, ?, 768), dtype=float32, device=/replica:0/task:0/device:GPU:0), dense_shape=Tensor("gradients/cond/Switch_1_grad/cond_grad/Shape:0", shape=(3,), dtype=int32, device=/replica:0/task:0/device:GPU:0))
More specifically: Incompatible CompositeTensor TypeSpecs: type=IndexedSlicesSpec str=IndexedSlicesSpec(TensorShape([Dimension(None), Dimension(None), Dimension(768)]), tf.float32, tf.int64, tf.int64, TensorShape([Dimension(None)])) vs. type=IndexedSlicesSpec str=IndexedSlicesSpec(TensorShape([Dimension(None), Dimension(None), Dimension(768)]), tf.float32, tf.int64, tf.int32, TensorShape([Dimension(None)]))
Entire first structure:
.
Entire second structure:
.