CDAN+DSBN about dsbn HOT 3 CLOSED

Hi, I haven't conducted experiments on CDAN.It seems that target bn won't update properly. Here are the tips to check.

First, use pretrained model(e.g. ImageNet, ...) and initialize each bn with the same value.
Second, check the magnitude of the gradients of target bn parameter. If the conditional adversarial loss is too small or too large compared to source bn parameter, the bn parameter won't update properly. In most case, use low learning rate on backbone with SGD will help. Also, you can consider to change discriminator output of sigmoid to softmax with 2 classes.
Third, adjust balance hyper-parameter λ (adjust source classifier and discriminator) or scheduling parameters if you use other datasets.

Good luck with your project :)

from dsbn.

Hi, thanks for your reply. In fact, I have used some tips you mentioned.

First, I used pretrained model on ImageNet, and used the dbsn.py and resnetdbsn.py which are from your respository to take place of the network.py in CDAN.
Second, I used the conditional adversarial loss in default setting same as CDAN, and the corresponding hyper-parameter λ is setted to 1 in default. Also, I used the default low learning rate (0.001) in target bn with SGD same as other layers (i.e., convolutional layers).
In short, I just utilized the dsbn layer to take place of the bn layer in CDAN without any other unnecessary change.

Just as you said, I also think the refine parameters in target bn cannot be updated properly. Dose it mean the model need a other loss function to help the target bn update properly?

from dsbn.

I think you need to check the magnitude of the gradients of bn parameters. If the target bn gradients are low, try to change domain discriminator output of sigmoid to softmax with 2 classes, which may provide stable gradients.
Also you can add additional loss. I think psuedo-label based recent algorithms work well with DSBN.

from dsbn.