frankfqchen / awesome-very-deep-learning Goto Github PK

awesome-very-deep-learning is a curated list for papers and code about implementing and training very deep neural networks.

Value Iteration Networks are very deep networks that have tied weights and perform approximate value iteration. They are used as an internal (model-based) planning module.

• Value Iteration Networks (2016) [original code], introduces VINs (Value Iteration Networks). The author shows that one can perform value iteration using iterative usage of convolutions and channel-wise pooling. It is able to generalize better in environments where a network needs to plan. NIPS 2016 best paper.

Densely Connected Convolutional Networks are very deep neural networks consisting of dense blocks. Within dense blocks, each layer receives the the feature maps of all preceding layers. This leverages feature reuse and thus substantially reduces the model size (parameters).

• Densely Connected Convolutional Networks (2016) [original code], introduces DenseNets and shows that it outperforms ResNets in CIFAR10 and 100 by a large margin (especially when not using data augmentation), while only requiring half the parameters.

0. Authors' Caffe Implementation
1. Authors' more memory-efficient Torch Implementation.
2. Tensorflow Implementation by Yixuan Li.
3. Tensorflow Implementation by Laurent Mazare.
4. Lasagne Implementation by Jan Schlüter.
5. Keras Implementation by tdeboissiere.
6. Keras Implementation by Roberto de Moura Estevão Filho.
7. Chainer Implementation by Toshinori Hanya.
8. Chainer Implementation by Yasunori Kudo.
9. PyTorch Implementation (including BC structures) by Andreas Veit
10. PyTorch Implementation

Deep Residual Networks are a family of extremely deep architectures (up to 1000 layers) showing compelling accuracy and nice convergence behaviors. Instead of learning a new representation at each layer, deep residual networks use identity mappings to learn residuals.

• Aggregated Residual Transformation for Deep Neural Networks (2016), introduces ResNeXt, which aggregates a set of transformations within a a res-block. It achieved the 2nd place on ILSVRC16.
• Residual Networks of Residual Networks: Multilevel Residual Networks (2016), adds multi-level hierarchical residual mappings and shows that this improves the accuracy of deep networks
• Wide Residual Networks (2016) [orginal code], studies wide residual neural networks and shows that making residual blocks wider outperforms deeper and thinner network architectures
• Swapout: Learning an ensemble of deep architectures (2016), improving accuracy by randomly applying dropout, skipforward and residual units per layer
• Deep Networks with Stochastic Depth (2016) [original code], dropout with residual layers as regularizer
• Identity Mappings in Deep Residual Networks (2016) [original code], improving the original proposed residual units by reordering batchnorm and activation layers
• Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning (2016), inception network with residual connections
• Deep Residual Learning for Image Recognition (2015) [original code], original paper introducing residual neural networks

0. Torch by Facebook AI Research (FAIR), with training code in Torch and pre-trained ResNet-18/34/50/101 models for ImageNet: blog, code
1. Torch, CIFAR-10, with ResNet-20 to ResNet-110, training code, and curves: code
2. Lasagne, CIFAR-10, with ResNet-32 and ResNet-56 and training code: code
3. Neon, CIFAR-10, with pre-trained ResNet-32 to ResNet-110 models, training code, and curves: code
4. Neon, Preactivation layer implementation: code
5. Torch, MNIST, 100 layers: blog, code
6. A winning entry in Kaggle's right whale recognition challenge: blog, code
7. Neon, Place2 (mini), 40 layers: blog, code
8. Tensorflow with tflearn, with CIFAR-10 and MNIST: code
9. Tensorflow with skflow, with MNIST: code
10. Stochastic dropout in Keras: code
11. ResNet in Chainer: code
12. Stochastic dropout in Chainer: code
13. Wide Residual Networks in Keras: code
14. ResNet in TensorFlow 0.9+ with pretrained caffe weights: code
15. ResNet in PyTorch: code

In addition, this [code] (https://github.com/ry/tensorflow-resnet) by Ryan Dahl helps to convert the pre-trained models to TensorFlow.

Highway Networks take inspiration from Long Short Term Memory (LSTM) and allow training of deep, efficient networks (with hundreds of layers) with conventional gradient-based methods

• Recurrent Highway Networks (2016) [original code], introducing recurrent highway networks, which increases space depth in recurrent networks
• Training Very Deep Networks (2015), introducing highway neural networks

0. Lasagne: code
1. Caffe: code
2. Torch: code
3. Tensorflow: blog, code

Theories in very deep learning concentrate on the ideas that very deep networks with skip connections are able to efficiently approximate recurrent computations (similar to the recurrent connections in the visual cortex) or are actually exponential ensembles of shallow networks

• The Shattered Gradients Problem: If resnets are the answer, then what is the question? argues that gradients of very deep networks resemble white noise (thus are harder to optimize). Resnets are more resistant to shattering (decaying sublinearly)
• Skip Connections as Effective Symmetry-Breaking hypothesizes that ResNets improve performance by breaking symmetries
• Highway and Residual Networks learn Unrolled Iterative Estimation, argues that instead of learning a new representation at each layer, the layers within a stage rather work as an iterative refinement of the same features.
• Demystifying ResNet, shows mathematically that 2-shortcuts in ResNets achieves the best results because they have non-degenerate depth-invariant initial condition numbers (in comparison to 1 or 3-shortcuts), making it easy for the optimisation algorithm to escape from the initial point.
• Wider or Deeper? Revisiting the ResNet Model for Visual Recognition, extends results from Veit et al. and shows that it is actually a linear ensemble of subnetworks. Wide ResNet work well, because current very deep networks are actually over-deepened (hence not trained end-to-end), due to the much shorter effective path length.
• Residual Networks are Exponential Ensembles of Relatively Shallow Networks, shows that ResNets behaves just like ensembles of shallow networks in test time. This suggests that in addition to describing neural networks in terms of width and depth, there is a third dimension: multiplicity, the size of the implicit ensemble
• Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex, shows that ResNets with shared weights work well too although having fewer parameters