How to train with custom datasets ? about dist_kd HOT 7 OPEN

I could fix the error by by changing the teacher model name in 'configs/strategies/distill/resnet_dist.yaml' from 'tv_resnet34' to 'resnet34'.
Now the student model trains well.

But I don't know how to improve the teacher model accuracy

from dist_kd.

Dear @ManuBN786 ,

Sorry for the late reply. Have you tried your dataset and training settings on your training framework or other frameworks?

from dist_kd.

Yes on a resnet50 from pytorch, it give a validation accuracy of 0.93.

I dont know how using DSIT_KD the validation accuracy is so poor

from dist_kd.

One bug I can find is that your training uses input images with 384x384 resolution, but the resolution in our framework is set to 224 with hard code. (see build_train_transforms and build_val_transforms in https://github.com/hunto/image_classification_sota/blob/main/lib/dataset/transform.py)

You should manually change all the 224 to 384 at L21, L32, and L61; and change 256 to 440 at L60.

from dist_kd.

Ok. Thanks for letting me know.

from dist_kd.

I did all of the above mentioned for image size 384, but I still get a very low accuracy for the teacher.

from dist_kd.

It's difficult for me to identify the differences between this repo and the example code by pytorch. If you want to use DIST KD in your project, I think the easiest way is to add KD code in our existing and valid code (You just need to initialize a pretrained teacher, compute its outputs wrt the batch input, and compute and backward the KD loss).

from dist_kd.