Inconsistent metrics about ditto HOT 4 CLOSED

I think the commands are okay except that you need to add the --save_model flag in the training. With the flag on, we will have a file named "catalogo_lm=distilbert_da=None_dk=None_su=False_size=None_id=0_dev.pt" which contains the checkpoint with the highest validation F1.

I ran

mv catalogo_lm=distilbert_da=None_dk=None_su=False_size=None_id=0_dev.pt checkpoints/catalogo.pt

then your command of running the matcher. I got

cat output/output_catalogo.jsonl | grep '"match": "1"' | wc -l 
2573
cat output/output_catalogo.jsonl | grep '"match": "0"' | wc -l 
2715

I also eye-balled some of the prediction results and they seemed to be correct.

from ditto.

I have run now and metrics it seems to make sense now. But... If I didn't save the model previously... What model did matcher.py process load?

from ditto.

I think it either initializes the model randomly or loads the pre-trained model (not optimized). We should just make it throw an exception. Thanks for pointing this out!

from ditto.

Great! You can check #6

from ditto.