Scores for LayoutLMv2 not matching with paper result about transformers-tutorials HOT 3 CLOSED

Hi,

I also experienced this, results can slightly differ between different runs. This is logical, as neural networks are not deterministic (unless using the same seed). However, running the notebook several times, I got the following results:

'overall_precision': 0.798961937716263, 'overall_recall': 0.8350813743218807, 'overall_f1': 0.8166224580017684

Which is (more or less) in line with the paper. The differences can be explained by the use of a different OCR engine (I'm using PyTesseract, whereas the original model used Azure Cognitive Services). You could also try with the OCR annotations already provided by the FUNSD dataset itself.

Also, the slight differences could be explained by the training hyperparameters (optimizer, learning rate, batch size, number of epochs, etc.)

Update: as can be seen here, they used the default HuggingFace Trainer, with the following additional hyperparameters:

--max_steps 1000 \
--warmup_ratio 0.1 \
--fp16

from transformers-tutorials.

I've uploaded a new notebook, which uses the HuggingFace Trainer and uses the same hyperparameters as the original implementation. Now I get:

'test_overall_precision': 0.8190854870775348, 'test_overall_recall': 0.8268941294530858, 'test_overall_f1': 0.8229712858926342'

from transformers-tutorials.

Thanks a lot. I will try out the new notebook once.

from transformers-tutorials.