AlexNet finetune on Office-31 about transferlearning HOT 18 CLOSED

Hey, I managed to get ~61% accuracy using Tensorflow on the A->W task, which I believe is good enough. I wrote a blog post about the tricks
TF code here: https://github.com/qinenergy/alexnet-finetune-office-31

Here are a few things I noticed that might effect performance:

Conv paddings: They are defined differently in TF and Caffe which can hurt performance when loading pretrained caffe weights.
lrn layer: Since I use pretrained caffe weights, I have to use lrn layers with the same parameters.
Data preprocessing: cv2.resize(256) -> randomcrop(227) I found that following the eact pipeline boosts the performance. I mean exact, use cv2, even tf.image could hurt performance.
Lr setting: I am not sure whether this helps, but I followed the eact lr settings of the GRL paper, which means 1x for pretrained kernels, 2x for pretrained biases, 10x for fc(31), 20x for fc(31) bias.

I don’t use PyTorch, so I am not sure the problems I encountered are the same as yours. Hope this helps.

from transferlearning.

@qinenergy Thank you for your reply! I will try your parameter setting to see what results I get. Keep in touch.

from transferlearning.

@qinenergy Could you please explain to me why your network has 9 layers? AlexNet has 8 layers with the final layers as fc6, fc7, and fc8.

from transferlearning.

@jindongwang IMHO, fc(128) is bottleneck layer, many DA papers using bottleneck layer in finetune.

from transferlearning.

@wogong But it is only the results of train on source, and test on target. There is no adaptation. Why bottleneck?

from transferlearning.

@jindongwang

emmm, my mistake. I thought this is talking about adaptation. I agree that there is no need for bottleneck layer in source-only baseline.

from transferlearning.

@jindongwang Hey, I agree with you that it is unnecessary to have a bottleneck layer in the source only case.

from transferlearning.

@qinenergy @wogong Then it's mysterious for the Office dataset. We can only consider using other popular datasets. This dataset is so awkward.

from transferlearning.

@jindongwang Have you tried Long's Xlearn repo https://github.com/thuml/Xlearn for reproducing the alexnet finetune results?
They use very different data pre-process code, like custom defined crop in the PyTorch part.
I am working on domain adaptation using PyTorch right now and come across the same problem as you. Nearly all current algorithms cannot get claimed accuracy on Office dataset using PyTorch implementations.

from transferlearning.

@wogong Long's data processing step can only help a little, which can be ignored.
It looks like no one can successfully reimplement the results of AlexNet and ResNet using Pytorch.

from transferlearning.

@jindongwang I use Resize(227) instead of Resize(256) and RandomResizedCrop(227) and get ~0.51 acc, which is really weird. I am not sure the reason but clearly data processing effect can not be ignored.

from transferlearning.

@wogong That's much better than what I got. Could you please modify the code in this repo to replace with your parameter settings?

from transferlearning.

@jindongwang I tried but failed to get the code run correctly. 😂
You can try my released repo https://github.com/wogong/pytorch-office_finetune.

from transferlearning.

BTW, I found this link in PyTorch forum. https://discuss.pytorch.org/t/pytorch-alexnet-not-as-good-as-original-alexnet-implementation/16680

Maybe the problem lies on torchvision's pretrained AlexNet model.

from transferlearning.

@wogong I also couldn't run your code successfully...
I know the LRN thing in AlexNet, but it seems doesn't help. A lot of people have demonstrated that LRN is not helping.
I tried your resize 227, it really works. Now my accuracy is above 50%.
I do not understand your code: why are there train-eval and test-eval dataset in loading the data?
And, have you tried ResNet?

from transferlearning.

@jindongwang emmm, I have not clear my code, so it may be confusing. src_eval and tgt_train is unnecessary. I'll try ResNet later.

from transferlearning.

@wogong @qinenergy I think we are making mistakes about the results from other papers. The evaluation protocol is different.
According to existing work (DDC, DAN, JAN, RTN, DCORAL etc.), the accuracy of AlexNet and ResNet are 61% and 80%. However, they are using different protocols: they take 20 samples per class (or 8 for dslr) as the source and target domains (we call it down-sample). Ours use the whole domain data (we call it full-training). This is quite different.
Actually, almost all the deep transfer learning methods are following this down-sampled protocol. Some research used the full-training protocol, but they just copy the results from the down-sample papers. Finally, they confuse us.

from transferlearning.

@wogong @qinenergy I checked the results again. They are using the full-training protocol. Still, we can't reproduce the results. Maybe it's time to give up Office.

from transferlearning.