Evaluation Script about compressai HOT 10 CLOSED

Glad to have been of use.

from compressai.

Hi Pascal, right the format it a bit different, the evaluation expects just the state dict whereas the training script also saves training information (loss value, optimizer state, etc..) as you can see from the print. I'll update the evaluation script to accept checkpoints from the training script. In the mean time, it should work if you extract the state_dict key in the checkpoint from the train example.

from compressai.

Hi Jean,

Thank you for your reply.
I extracted the state_dict key from the checkpoint and feed it to compressai.utils.eval_model. But after training the net needs to be updated or it raises errors. In the end, I managed to make it work.

However when I downloaded the corresponding pretrained tar file to test compressai.utils.eval_model.
https://compressai.s3.amazonaws.com/models/v1/bmshj2018-factorized-prior-3-5c6f152b.pth.tar

I got some errors:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/udd/dbacchus/Codes/CompressAI/compressai/utils/eval_model/__main__.py", line 301, in <module>
    main(sys.argv[1:])
  File "/udd/dbacchus/Codes/CompressAI/compressai/utils/eval_model/__main__.py", line 277, in main
    model = load_func(*opts, run)
  File "/udd/dbacchus/Codes/CompressAI/compressai/utils/eval_model/__main__.py", line 145, in load_checkpoint
    return architectures[arch].from_state_dict(torch.load(checkpoint_path)).eval()
  File "/udd/dbacchus/Codes/CompressAI/compressai/models/priors.py", line 168, in from_state_dict
    net.load_state_dict(state_dict)
  File "/udd/dbacchus/Codes/CompressAI/compressai/models/priors.py", line 105, in load_state_dict
    super().load_state_dict(state_dict)
  File "/udd/dbacchus/env_py3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for FactorizedPrior:
	Missing key(s) in state_dict: "entropy_bottleneck._matrix0", "entropy_bottleneck._bias0", "entropy_bottleneck._factor0", "entropy_bottleneck._matrix1", "entropy_bottleneck._bias1", "entropy_bottleneck._factor1", "entropy_bottleneck._matrix2", "entropy_bottleneck._bias2", "entropy_bottleneck._factor2", "entropy_bottleneck._matrix3", "entropy_bottleneck._bias3", "entropy_bottleneck._factor3", "entropy_bottleneck._matrix4", "entropy_bottleneck._bias4". 
	Unexpected key(s) in state_dict: "entropy_bottleneck._biases.0", "entropy_bottleneck._biases.1", "entropy_bottleneck._biases.2", "entropy_bottleneck._biases.3", "entropy_bottleneck._biases.4", "entropy_bottleneck._factors.0", "entropy_bottleneck._factors.1", "entropy_bottleneck._factors.2", "entropy_bottleneck._factors.3", "entropy_bottleneck._matrices.0", "entropy_bottleneck._matrices.1", "entropy_bottleneck._matrices.2", "entropy_bottleneck._matrices.3", "entropy_bottleneck._matrices.4".

I found differences in both state_dict,, especially in the item's order.

from compressai.

Yes you need to update the model to compute the CDFs based on the trained parameters of the entropy bottleneck(s) (as per the doc here: https://interdigitalinc.github.io/CompressAI/tutorial_train.html#updating-the-model).

Regarding the error from the pretrained model, can you please provide the compressai version you are using?

from compressai.

I'm using the latest version v1.1.1.

from compressai.

ok thx, can you provide a short script so I can reproduce the error?

from compressai.

OK sure, it's quite simple really.

I just trained a model (bmshj2018-factorized) on my own dataset.
python3 train.py -d $DATASET --epochs 100 -lr 1e-4 --batch-size 16 --cuda --save

Then I downloaded a pretrained tar file as indicated in the documentation.
https://compressai.s3.amazonaws.com/models/v1/bmshj2018-factorized-prior-3-5c6f152b.pth.tar

Finally I tried to run the compressai.utils.eval_model script
python3 -m compressai.utils.eval_model checkpoint $DATASET -a bmshj2018-factorized -p bmshj2018-factorized-prior-3-5c6f152b.pth.tar

from compressai.

Got it thanks!
So we changed the state dict format a few months ago to support multi gpu. Thanks for reporting this, we should convert the models in both checkpoint and pretrained mode.

Side note, you actually don't have to download manually the pretrained weights but can use this command python3 -m compressai.utils.eval_model pretrained ...

from compressai.

I also encountered the same error, can you explain how to solve it?

from compressai.

@jbegaint @PascalBACCHUS

from compressai.