Git Product home page Git Product logo

cifar's Introduction

CIFAR10 Sandbox

Mixup

200 epochs (Learning rate divided by 10 at epoch 100 and 150)

Method Acc/Error (%) Config Parameters
Baseline 94.86/5.14 baseline/baseline.yaml
Mixup 96.01/3.99 mixup/200/mixup.yaml α=1
Manifold Mixup 96.10/3.90 mixup/200/manifold_mixup01.yaml α=2
layers=[0,1]
Manifold Mixup 96.01/3.99 mixup/200/manifold_mixup012.yaml α=2
layers=[0,1,2]
Cutmix 96.20/3.80 mixup/200/cutmix.yaml α=1
Manifold Cutmix 96.06/3.94 mixup/200/manifold_cutmix_a1.yaml α=1
layers=[0,1,2]
Manifold Cutmix 95.70/4.30 mixup/200/manifold_cutmix_a2.yaml α=2
layers=[0,1,2]

600 epochs (Learning rate divide by 10 at epoch 300 and 450)

Method Acc/Error (%) Config Parameters
Baseline 95.40/4.60 baseline/baseline_600.yaml
Mixup 96.59/3.41 mixup/600/mixup.yaml α=1
Manifold Mixup 96.86/3.14 mixup/600/manifold_mixup_a2.yaml α=2
layers=[0,1,2]
Manifold Mixup 96.63/3.37 mixup/600/manifold_mixup_a1.yaml α=1
layers=[0,1,2]
Cutmix 96.76/3.24 mixup/600/cutmix.yaml α=1
Manifold Cutmix 96.53/3.47 mixup/600/manifold_cutmix_a1.yaml α=1
layers=[0,1,2]
Manifold Cutmix 96.43/3.57 mixup/600/manifold_cutmix_a2.yaml α=2
layers=[0,1,2]

1200 epochs (Learning rate divided by 10 at epoch 400 and 800)

Method Acc/Error (%) Config Parameters
Baseline 95.59/4.41 baseline/baseline_1200.yaml
Mixup 96.85/3.15 mixup/1200/mixup.yaml α=1
Manifold Mixup 97.19/2.81 mixup/1200/manifold_mixup.yaml α=2
layers=[0,1,2]

Training with GAN Data

  • Images generated from conditional BigGAN.

200 epochs (Learning rate divided by 10 at epoch 100 and 150)

% Generated Data Acc/Error (%) Config
100% 66.23/33.77 gan/gan_100.yaml
50% 92.34/7.66 gan/gan_50.yaml
25% 93.67/6.33 gan/gan_25.yaml
10% 94.33/5.67 gan/gan_10.yaml
0% 94.86/5.14 baseline/baseline.yaml

Augmentations

  • The standard augmentations are random horizonal flips, random translations and normalization. Addition augmentations are applied after the random flip and translation while normalization is always applied last. Cutout is applied after Autoaugment/RandAugment when used together.

200 epochs (Learning rate divided by 10 at epoch 100 and 150)

Method Acc/Error (%) Config Parameters
Baseline 94.86/5.14 baseline/baseline.yaml
Cutout 95.88/4.12 augment/200/cutout.yaml cutout=16x16
AutoAugment 95.90/4.10 augment/200/autoaugment.yaml
AutoAugment + Cutout 96.31/3.69 augment/200/autoaugment_cutout.yaml cutout=16x16
RandAugment 95.02/4.98 augment/200/randaugment_n3m5.yaml n=3
m=5
RandAugment 94.93/5.07 augment/200/randaugment_n3m4.yaml n=3
m=4
RandAugment 94.37/5.63 augment/200/randaugment_n3m2.yaml n=3
m=2
RandAugment 95.65/4.35 augment/200/randaugment_n2m5.yaml n=2
m=5
RandAugment 95.50/4.50 augment/200/randaugment_n2m6.yaml n=2
m=6
RandAugment + Cutout 95.64/4.36 augment/200/randaugment_cutout.yaml n=2
m=5
cutout=16x16
RandAugment (w/ Cutout) 95.63/4.37 augment/200/randaugment_include_cutout.yaml n=2
m=5
GridMask 95.59/4.41 augment/200/gridmask_8_32_r04.yaml minD=8
maxD=32
r=0.4
GridMask 95.88/4.12 augment/200/gridmask_16_32_r04.yaml minD=16
maxD=32
r=0.4
GridMask 95.78/4.22 augment/200/gridmask_16_40_r04.yaml minD=16
maxD=40
r=0.4
GridMask 95.80/4.20 augment/200/gridmask_16_32_r05.yaml minD=16
maxD=32
r=0.5
GridMask 95.69/4.31 augment/200/gridmask_16_32_r03.yaml minD=16
maxD=32
r=0.3
Autoaugment + GridMask 96.18/3.82 augment/200/autoaugment_gridmask_16_32_r04.yaml minD=16
maxD=32
r=0.4
Autoaugment + GridMask 95.95/4.05 augment/200/autoaugment_gridmask_16_32_r03.yaml minD=16
maxD=32
r=0.3
AugMix 95.63/4.37 augment/200/augmix_w3_d3_s3.yaml width=3
depth=3
severity=3
AugMix 95.53/4.47 augment/200/augmix_w3_d3_s5.yaml width=3
depth=3
severity=5

600 epochs (Learning rate divide by 10 at epoch 300 and 450)

Method Acc/Error (%) Config Parameters
Baseline 95.40/4.60 baseline/baseline_600.yaml
AutoAugment + Cutout 97.09/2.91 augment/600/autoaugment_cutout.yaml cutout=16x16
RandAugment + Cutout 96.5/3.5 augment/600/randaugment_cutout.yaml n=2
m=5
cutout=16x16
RandAugment (w/ Cutout) 96.58/3.42 augment/600/randaugment_include_cutout.yaml n=2
m=5
Autoaugment + GridMask 97.02/2.98 augment/600/autoaugment_gridmask_16_32_r04.yaml minD=16
maxD=32
r=0.4
Autoaugment + GridMask 96.81/3.19 augment/600/autoaugment_gridmask_16_32_r03.yaml minD=16
maxD=32
r=0.3
AugMix 96.33/3.67 augment/600/augmix_w3_d3_s3.yaml width=3
depth=3
severity=3
  • AutoAugment shows better results on CIFAR compared to RandAugment which appears to make the augmentations too strong for the size of the model without proper hyperparameter tuning. The hyperparameters however offer better flexiblity which should benefit a wider variety of datasets compared to AutoAugment which is tuned specifically on a target dataset.

Other Regularization

200 epochs (Learning rate divided by 10 at epoch 100 and 150)

Method Acc/Error (%) Config Parameters
Baseline 94.86/5.14 baseline/baseline.yaml
Label Smoothing 94.69/5.31 other/smoothing_01.yaml ε=0.1

Combinations

600 Epochs (Cosine Annealing Schedule)

Method Acc/Error (%) Config Parameters
Baseline 95.06/4.94 baseline/baseline_600_cos.yaml
AutoAugment + Cutout 97.06/2.94 combination/autoaugment_cutout.yaml cutout=16x16
AutoAugment + Cutout + Label Smoothing - combination/autoaugment_cutout_smoothing.yaml cutout=16x16
ε=1
AutoAugment + Cutout + Mixup 97.25/2.75 combination/autoaugment_cutout_mixup.yaml cutout=16x16
α=1
AutoAugment + Cutout + Mixup + Label Smoothing 97.26/2.74 combination/aa_cutout_mixup_smoothing.yaml cutout=16x16
α=1
ε=0.1
AutoAugment + Cutout + Manifold Mixup 97.05/2.95 combination/autoaugment_cutout_manifold.yaml cutout=16x16
α=2
layers=[0,1,2]
AutoAugment + Cutout + Manifold Mixup + Label Smoothing - combination/aa_cutout_manifold_smoothing.yaml cutout=16x16
α=2
layers=[0,1,2]
ε=0.1
AutoAugment + Cutmix - combination/autoaugment_cutmix.yaml α=1

To Do

  • DropBlock, StochDepth
  • Truncated z sampling in GAN
  • CIFAR100
  • Different architectures
  • Architecture modifications (Shake-Shake, ShakeDrop, etc)
  • Further investigate AugMix and GridMask

References

cifar's People

Contributors

bwconrad avatar

Stargazers

avatar

Watchers

avatar avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.