Git Product home page Git Product logo

tricks-of-semi-superviseddeepleanring-pytorch's Introduction

Tricks of Semi-supervised Deep Leanring --Pytorch

The repository implements following semi-supervised deep learning methods:

  • PseudoLabel 2013: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks (ICMLW 2013)

  • PI&Tempens: Temporal Ensembling for Semi-Supervised Learning (ICLR 2017)

  • MeanTeacher: Mean Teachers are better Role Models (NIPS 2017)

  • VAT: Virtual Adversarial Training: A Regularization Method for Supervised and Semi-supervised Learning (TPAMI 2018)

  • ICT: Interpolation Consistency Training for Semi-supervised Learning (IJCAI 2019)

  • MixMatch: A Holistic Approach to Semi-Supervised Learning (NIPS 2019)

  • FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020)

This repository was created for my blog 半监督深度学习训练和实现小Tricks. Therefore the hyper-parameters are set for fair comparision, rather than performance.

The environment:

  • Ubuntu 16.04 + CUDA 9.0

  • Python 3.6.5:: Anaconda

  • PyTorch 0.4.1 and torchvision 0.2.1

To run the code:

The script run.sh includes some examples. You can try it as follow:

bash run.sh [gpu_id]

Some experimental results:

I haven't run all models in this repository. Some results of this repo. are shown in results directory. And the following results came from this repository and the old codes which this repo. built on.

The following table shows the error rates of the CIFAR10 experiment with 4000 labeled training samples. The parameter settings are the same with the examples in run.sh.

iPseudoLabel2013 ePseudoLabel2013 MeanTeacher MixMatch iFixMatch
orginal 12.31 6.24 4.26
v1 20.03 12.03 10.59 6.70 6.63
v2 15.82 10.82 9.46 6.89 6.44
iTempens eTempens PI ICT* VAT
orginal 12.16 13.20 7.29 11.36
v1 10.98 10.74 14.11 7.12 13.84
v2 13.53 10.24 12.89 6.74 12.67
eMixPseudoLabelv1 eMixPseudoLabelv2
soft 7.30 7.08
hard 7.33 7.20

Notes:

  • The MeanTeacher is the first model of the repository. So the hyper-parameters actually have been tuned for MeanTeacher.

  • My ICT is different from original one. The main difference is the unsupervised loss for unlabeled data.

  • eMixPseudoLabel is ePseudoLabel2013 with MixUp.

  • Instead of KL Divergence, my VAT uses MSE which bring more performance improvement.

tricks-of-semi-superviseddeepleanring-pytorch's People

Contributors

ibelievecjm avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

tricks-of-semi-superviseddeepleanring-pytorch's Issues

etempensv1/2 got RuntimeError: Index put requires the source and destination dtypes match, got Long for the destination and Float for the source

Hi,
When I run etempensv1 I got this error message:
pytorch version : 1.7.1
Files already downloaded and verified
Files already downloaded and verified
Tempens-v1 with epoch pseudo labels
------ Training epochs: 0 ------
/home/xps8500/anaconda3/envs/himp/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Traceback (most recent call last):
File "main.py", line 165, in
run(config)
File "main.py", line 160, in run
trainer.loop(config.epochs, *loaders, scheduler=scheduler)
File "/home/xps8500/ssdlp/trainer/eTempensv1.py", line 125, in loop
self.train(train_data, self.print_freq)
File "/home/xps8500/ssdlp/trainer/eTempensv1.py", line 99, in train
return self.train_iteration(data_loader, print_freq)
File "/home/xps8500/ssdlp/trainer/eTempensv1.py", line 56, in train_iteration
self.epoch_pslab[idxs] = outputs.clone().detach()
RuntimeError: Index put requires the source and destination dtypes match, got Long for the destination and Float for the source.

The error message from etempensv2 is:

pytorch version : 1.7.1
Files already downloaded and verified
Files already downloaded and verified
Tempens-v2 with epoch pseudo labels
------ Training epochs: 0 ------
/home/xps8500/anaconda3/envs/himp/lib/python3.6/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
return torch._C._cuda_getDeviceCount() > 0
/home/xps8500/anaconda3/envs/himp/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
"https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
Traceback (most recent call last):
File "main.py", line 165, in
run(config)
File "main.py", line 160, in run
trainer.loop(config.epochs, *loaders, scheduler=scheduler)
File "/home/xps8500/ssdlp/trainer/eTempensv2.py", line 127, in loop
self.train(label_data, unlab_data, self.print_freq)
File "/home/xps8500/ssdlp/trainer/eTempensv2.py", line 101, in train
return self.train_iteration(label_loader, unlab_loader, print_freq)
File "/home/xps8500/ssdlp/trainer/eTempensv2.py", line 55, in train_iteration
uloss = self.mse_loss(unlab_outputs, iter_unlab_pslab)
File "/home/xps8500/ssdlp/utils/loss.py", line 34, in mse_with_softmax
return F.mse_loss(F.softmax(logit1,1), F.softmax(logit2,1))
File "/home/xps8500/anaconda3/envs/himp/lib/python3.6/site-packages/torch/nn/functional.py", line 1512, in softmax
ret = input.softmax(dim)
RuntimeError: "softmax_lastdim_kernel_impl" not implemented for 'Long'

Thanks!

Questions on FixMatch

Hi, thanks for sharing. I have been playing with FixMatch lately. Do you find eFixMatch perform better than iFixMatch? The comparison doesn't show up in the readme.

Also, do you have experience deploying FixMatch on imbalanced data? I can't make it work so far. FixMatch performs equivalent to or worse than a naive supervised learning method.

I created an issue on google's repo: google-research/fixmatch#52. Welcome any comments.

FixMatch for single class

I try to apply FixMatch on one class data.
For the unsupervised loss part, I modified the code like that.

self.uce_loss   = torch.nn.BCEWithLogitsLoss(reduction='none')
with torch.no_grad():
    woutputs = self.model(weak_u)
    woutputs = torh.sigmoid(woutput)
      
           
mask = woutputs .ge(self.threshold).float()
soutputs = self.model(strong_u)
uloss  = torch.mean(mask* self.uce_loss(soutputs, mask) )

Is that correct?

clarification on pseudo

Nice repository! What is the difference between epseudo and ipseudo algortihms, and the difference between versions?
Thanks in advance.

reimplement performance

it is a really nice work for semi-surpervised leaning. would you mind provide the performance of your code compared with paper performance?

test and dev set

I found that there are bugs in the code, and the data in the dev set may be mixed in the test set !

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.