UnboundLocalError about local variable 'this_epoch_val_metric' in Allennlp trainer.py about declutr HOT 7 CLOSED

I confirmed that this was a bug In allennlp>1.20. See here. Installing "allennlp>=1.1.0, <1.2.0" should fix your issue. Please re-open the issue if it does not.

from declutr.

Do you mind posting the full stack trace? And let me know what version of AllenNLP you are using?

If I had to guess, this error is actually coming from AllenNLP.

from declutr.

I am able to trigger this issue if I supply a non-zero "patience" argument to the trainer, without also supplying a "validation_data_path" argument. Is that what you are doing, perhaps?

from declutr.

Thanks for your reply, I added validation_data_path and it resolved, but in the model output it seems the validation_set was not used.... so your suggestion now is I should apply non-zero "patience" argument to the trainer?

from declutr.

You shouldn’t need to set validation_data_path or patience. You will notice that neither are set in the default configs in this repo.

from declutr.

xt_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.key.bias'], {}
2020-11-21 15:39:08,571 - INFO - allennlp.training.optimizers - Number of trainable parameters: 82170201
2020-11-21 15:39:08,574 - INFO - allennlp.common.util - The following parameters are Frozen (without gradient):
2020-11-21 15:39:08,576 - INFO - allennlp.common.util - The following parameters are Tunable (with gradient):
2020-11-21 15:39:08,576 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.embeddings.word_embeddings.weight
2020-11-21 15:39:08,576 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.embeddings.position_embeddings.weight
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.embeddings.token_type_embeddings.weight
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.embeddings.LayerNorm.weight
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.embeddings.LayerNorm.bias
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.self.query.weight
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.self.query.bias
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.self.key.weight
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.self.key.bias
2020-11-21 15:39:08,577 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.self.value.weight
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.self.value.bias
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.output.dense.weight
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.output.dense.bias
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.output.LayerNorm.weight
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.attention.output.LayerNorm.bias
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.intermediate.dense.weight
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.intermediate.dense.bias
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.output.dense.weight
2020-11-21 15:39:08,578 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.output.dense.bias
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.output.LayerNorm.weight
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.0.output.LayerNorm.bias
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.query.weight
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.query.bias
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.key.weight
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.key.bias
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.value.weight
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.self.value.bias
2020-11-21 15:39:08,579 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.output.dense.weight
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.output.dense.bias
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.output.LayerNorm.weight
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.attention.output.LayerNorm.bias
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.intermediate.dense.weight
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.intermediate.dense.bias
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.output.dense.weight
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.output.dense.bias
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.output.LayerNorm.weight
2020-11-21 15:39:08,580 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.1.output.LayerNorm.bias
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.self.query.weight
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.self.query.bias
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.self.key.weight
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.self.key.bias
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.self.value.weight
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.self.value.bias
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.output.dense.weight
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.output.dense.bias
2020-11-21 15:39:08,581 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.output.LayerNorm.weight
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.attention.output.LayerNorm.bias
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.intermediate.dense.weight
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.intermediate.dense.bias
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.output.dense.weight
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.output.dense.bias
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.output.LayerNorm.weight
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.2.output.LayerNorm.bias
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.self.query.weight
2020-11-21 15:39:08,582 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.self.query.bias
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.self.key.weight
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.self.key.bias
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.self.value.weight
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.self.value.bias
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.output.dense.weight
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.output.dense.bias
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.output.LayerNorm.weight
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.attention.output.LayerNorm.bias
2020-11-21 15:39:08,583 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.intermediate.dense.weight
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.intermediate.dense.bias
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.output.dense.weight
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.output.dense.bias
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.output.LayerNorm.weight
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.3.output.LayerNorm.bias
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.self.query.weight
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.self.query.bias
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.self.key.weight
2020-11-21 15:39:08,584 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.self.key.bias
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.self.value.weight
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.self.value.bias
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.output.dense.weight
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.output.dense.bias
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.output.LayerNorm.weight
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.attention.output.LayerNorm.bias
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.intermediate.dense.weight
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.intermediate.dense.bias
2020-11-21 15:39:08,585 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.output.dense.weight
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.output.dense.bias
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.output.LayerNorm.weight
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.4.output.LayerNorm.bias
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.self.query.weight
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.self.query.bias
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.self.key.weight
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.self.key.bias
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.self.value.weight
2020-11-21 15:39:08,586 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.self.value.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.output.dense.weight
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.output.dense.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.output.LayerNorm.weight
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.attention.output.LayerNorm.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.intermediate.dense.weight
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.intermediate.dense.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.output.dense.weight
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.output.dense.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.output.LayerNorm.weight
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.roberta.encoder.layer.5.output.LayerNorm.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.lm_head.bias
2020-11-21 15:39:08,587 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.lm_head.dense.weight
2020-11-21 15:39:08,588 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.lm_head.dense.bias
2020-11-21 15:39:08,588 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.lm_head.layer_norm.weight
2020-11-21 15:39:08,588 - INFO - allennlp.common.util - _text_field_embedder.token_embedder_tokens.transformer_model.lm_head.layer_norm.bias
2020-11-21 15:39:08,588 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.type = slanted_triangular
2020-11-21 15:39:08,588 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.cut_frac = 0.1
2020-11-21 15:39:08,588 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.ratio = 32
2020-11-21 15:39:08,588 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.last_epoch = -1
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.gradual_unfreezing = False
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.discriminative_fine_tuning = False
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.learning_rate_scheduler.decay_factor = 0.38
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.checkpointer.type = default
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.checkpointer.keep_serialized_model_every_num_seconds = None
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.checkpointer.num_serialized_models_to_keep = -1
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - trainer.checkpointer.model_save_interval = None
2020-11-21 15:39:08,589 - INFO - allennlp.common.params - summary_interval = 100
2020-11-21 15:39:08,590 - INFO - allennlp.common.params - histogram_interval = None
2020-11-21 15:39:08,590 - INFO - allennlp.common.params - batch_size_interval = None
2020-11-21 15:39:08,590 - INFO - allennlp.common.params - should_log_parameter_statistics = True
2020-11-21 15:39:08,590 - INFO - allennlp.common.params - should_log_learning_rate = False
2020-11-21 15:39:08,590 - INFO - allennlp.common.params - get_batch_num_total = None
2020-11-21 15:39:08,594 - INFO - allennlp.training.trainer - Beginning training.
2020-11-21 15:39:08,594 - INFO - allennlp.training.trainer - Epoch 0/0
2020-11-21 15:39:08,594 - INFO - allennlp.training.trainer - Worker 0 memory usage: 2.4G
2020-11-21 15:39:08,595 - INFO - allennlp.training.trainer - GPU 0 memory usage: 315M
2020-11-21 15:39:08,596 - INFO - allennlp.training.trainer - Training
0it [00:00, ?it/s]/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/data/dataset_readers/dataset_reader.py:377: UserWarning: Using multi-process data loading without setting DatasetReader.manual_multi_process_sharding to True.
Did you forget to set this?
If you're not handling the multi-process sharding logic within your _read() method, there is probably no benefit to using more than one worker.
UserWarning,
2020-11-21 15:39:08,670 - INFO - declutr.dataset_reader - Reading instances from lines in file at: /home/gnn_research/good_fellas/DeCLUTR/covid_preprocessed_data.csvt [00:00, ?it/s]
Truncation was not explicitely activated but max_length is provided a specific value, please use truncation=True to explicitely truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to truncation.
reading instances: 12181it [18:29, 10.98it/s] [18:29, 2.78it/s]
batch_loss: 2.0792, loss: 2.5238 ||: : 3045it [18:30, 2.74it/s]
2020-11-21 15:57:40,283 - INFO - allennlp.training.tensorboard_writer - Training | Validation
2020-11-21 15:57:40,284 - INFO - allennlp.training.tensorboard_writer - gpu_0_memory_MB | 314.692 | N/A
2020-11-21 15:57:40,284 - INFO - allennlp.training.tensorboard_writer - loss | 2.524 | N/A
2020-11-21 15:57:40,285 - INFO - allennlp.training.tensorboard_writer - worker_0_memory_MB | 2441.402 | N/A
2020-11-21 15:57:40,287 - CRITICAL - root - Uncaught exception
Traceback (most recent call last):
File "/home/gnn_research/miniconda3/envs/DeCLUTR/bin/allennlp", line 8, in
sys.exit(run())
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/main.py", line 34, in run
main(prog="allennlp")
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/commands/init.py", line 118, in main
args.func(args)
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/commands/train.py", line 119, in train_model_from_args
file_friendly_logging=args.file_friendly_logging,
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/commands/train.py", line 178, in train_model_from_file
file_friendly_logging=file_friendly_logging,
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/commands/train.py", line 242, in train_model
file_friendly_logging=file_friendly_logging,
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/commands/train.py", line 466, in _train_worker
metrics = train_loop.run()
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/commands/train.py", line 528, in run
return self.trainer.train()
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/training/trainer.py", line 966, in train
return self._try_train()
File "/home/gnn_research/miniconda3/envs/DeCLUTR/lib/python3.6/site-packages/allennlp/training/trainer.py", line 1082, in _try_train
self._learning_rate_scheduler.step(this_epoch_val_metric)
UnboundLocalError: local variable 'this_epoch_val_metric' referenced before assignment
srun: error: master-node: task 0: Exited with exit code 1

When I remove validation_data_path from declutr_small.jsonnet it gives me the above error, when add validation_path the error is resolved. The allennlp and allennlp-models versions are both 1.2.1.

from declutr.

Will you please try using a version of allennlp<1.2.0. Notice that this is specified in the setup.py file.

The correct version will be installed for you if you following the installation instructions.

allennlp-models is not a requirement of this repo, so you shouldn't need it.

from declutr.