fengjixuchui / how-to-learn-deep-learning Goto Github PK

A practical, top-down approach, starting with high-level frameworks with a focus on Deep Learning. It will enable you to create your own projects in deep learning. To invent new deep learning algorithms, also check the theory section.

Note: The key is to start building your own projects as soon as possible.

• Spend a week on codecademy.com and learn the python syntax, command line and git.
• Spend one to two weeks using Pandas and Scikit-learn on Kaggle problems using Jupyter Notebook, e.g. Titanic, House prices, and Iris. This gives you an overview of the machine learning mindset and workflow.
• Spend one month implementing models on cloud GPUs. Most beginners tend to start with Keras with a Tenorflow-backend or Pytorch using the FastAI library. Start with these Keras models or the official Pytorch tutorials. Also, check the resources from FastAI. The FastAI community if becoming the go-to place for people wanting to apply deep learning and share the state of the art techniques. Warning: don't get stuck taking online courses and start building as soon as possible.
• Spend one month recoding the core concepts in python numpy, including a The Method of Least Squares, Gradient Descent, Linear Regression, The Perceptron and a vanilla neural network. Andrew Trask's book and blog are great for this. This will help you reduce a lot of cognitive load down the line.

By reproducing papers you get a feel for doing actual work in deep learning. Reproduce a paper or building a project in the following four areas: vision, natural language processing, generative networks, and reinforcement learning/evolution.

Tips for reproducing papers:

• It takes 1-2 months to reproduce a student paper if you work full-time. It takes about 3 weeks to get clarity of the core concepts in each paper.
• Find github reimplementations, but avoid looking at repos using the same framework as you, since the temptation to copy-paste will be too high. Read all the repo issues, since the language is easier and you have a high chance of struggling with the same things.
• I often build the network in four steps. Start with one data example and make anything that runs. At this stage, it's okay if you only understood 20% of the paper and your network is buggy. Once you have something to experiment, it makes it a lot easier to add the remaining 80% of the knowledge. For the second stage, I start automating the training flow and try to overfit the network. Once you can overfit the model, I build a structure to create a network that can generalize. For the final part, I implement multiprocessing and optimize the CPU and GPU utilization.
• Discussions online are in general a good way to understand a paper better. Google is a good starting point, but also check reviewing sites such as openreview, Arxiv-sanity or Twitter search.
• Don't follow a step by step tutorial or MOOC. Instead, spend a few days scanning every MOOC and tutorial on the topic. This gives you an index of resources you can later dig deeper in. If you follow a step by step guide, you end up copy-pasting instead of learning anything.
• It's very cognitive demanding to learn deep learning. To feel a sense of progress, I'd recommend scheduling everything you do. Also, have a Pomodoro timer and block all news/notifications/social media.
• Most Q&A/forums will offer little help in solving your bugs. The best option is to document the problem you are facing in detail, then research all the unknowns. It often helps if you refactor your code and make small tests to try different assumptions about your model. I'd also suggest reaching out to the author of the paper you are reproducing. Again, if you reproduce student papers they are often happy to answer clarifying questions.
• Share your learning. It's a great way to learn, you gain exposure, and you help people in the same position as you.

Start with the practical approach, then you'll be familiar with many of the concepts when you start adding theoretical knowledge. I had 18 months of practical experience before I started digging into the theory. I believe the best deep learning theory curriculum is the Deep Learning Book by Ian Goodfellow and Yoshua Bengio and Aaron Courville.

I'm using Michael Nielsen's Anki method to process the material.

• I normally spend 6 hours reading 5 - 20 pages and finding YouTube tutorials to understand every concept. This includes doing exercises on Khan Academy and redoing derivations from lectures on Youtube.
• Then I spend 2 - 3 hours creating Anki cards. I have one deck for questions I can rehearse on the go, and another for exercises I need a pen and paper for.
• I spend another hour reviewing previous Anki cards and exercises.

I set aside three months to study the Deep Learning Book fulltime. The book is 700 pages, so aim to ankify 10 - 15 pages per day. The first few chapters are denser, so I began with 5 pages a day and gradually got closer to 15 pages.

Once you have a broad theoretical knowledge of deep learning, I'd recommend Ankifying ~100 papers in the field you want to build an edge in. I'm using OpenAI's curriculum and paper recommendations to get up to speed with the research in deep reinforcement learning.

• FastAI
• Keras Slack
• Distill Slack
• Pytorch

The best way to get a feel for the most interesting ideas in Machine Learning is Twitter and Arxiv-sanity. Here is my full list of people I follow on Twitter. These are my favorites: hardmaru, ilyasut, josephreisinger, math_rachel, mustafasuleymn, catherineols, dennybritz, ylecun, jtoy, brohrer, tkasasagi, jeremyjkun, jeffclune, danielgross, karoly_zsolnai, mortendahlcs, Reza_Zadeh, goodfellow_ian, fchollet, michael_nielsen, iamtrask, jeremyphoward, jackclarkSF, ch402, distillpub.

GPU access is key. If you want a fully configured development environment, Floydhub is perfect. If you have a GPU budget that is less than 100$/month I'd recommend colab.research.google.com. They offer free TPU instances for ~6h. Kaggle.com also offers free ~6h training sessions on Tesla P100s. If you have a low budget, yet need a lot of compute, I'd recommend paying 100$ for producthunt.com's subscription, which gives you $7.5K in AWS cloud credit.

But the best option is to create a deep learning startup and apply for cloud credit. You don't need more than a landing page and a company email to get $5K-100K in cloud credit. You can apply on Google's startup program or through one of their partners. AWS, Digital Ocean, Azure, and Alibaba also have cloud credit programs for startups.

If you have +$3K budget, building your own rig is a great option.

• Implementing Keras models (I started with TFlearn, but I'd highly recommend using Keras instead) (I mistakenly thought TFlearn was the official frontend. They have now made Keras the official front-end to Tensorflow.)
• Recoding the core concepts in python
• My first paper using CNN
• My first paper using LSTM
• My first paper using a GAN
• My first paper using RL/evolution
• On going via my twitter feed

• Per Harald Borgen
• Greg Brockman
• Andrew Ng
• Amid Fish
• Spinning Up by OpenAI
• Confession as an AI researcher

If you have suggestions/questions create an issue or ping me on Twitter.