gordicaleksa / pytorch-deepdream Goto Github PK

This repo contains a PyTorch implementation of the Deep Dream algorithm (:link: blog by Mordvintstev et al.).

It's got a full support for the command line usage and a Jupyter Notebook!

And it will give you the power to create these weird, psychedelic-looking images:

Not bad, huh?

I strongly suggest you start with the Jupyter notebook that I've created!

Note: it's pretty large, ~10 MBs, so it may take a couple of attempts to load it in the browser here on GitHub.

• What is DeepDream?
  • Image visualizations and experiments
  • Ouroboros video examples
  • DeepDream video examples
• Setup
• Usage
• Hardware requirements
• Learning material

In a nutshell the algorithm maximizes the activations of chosen network layers by doing a gradient ascent.

So from an input image like the one on the left after "dreaming" we get the image on the right:

Who would have said that neural networks had this creativity hidden inside? 🎨

Most of the original Deep Dream repos were written in Caffe and the ones written in PyTorch are usually really hard to read and understand. This repo is an attempt of making the cleanest DeepDream repo that I'm aware of + it's written in PyTorch! ❤️

Here are some examples that you can create using this code!

By using shallower layers of neural networks you'll get lower level patterns (edges, circles, colors, etc.) as the output:

Here the first 2 images came from ResNet50 and the last one came from GoogLeNet (both pretrained on ImageNet).

By using deeper network layers you'll get higher level patterns (eyes, snouts, animal heads):

The 1st and 3rd were created using VGG 16 (ImageNet) and the middle one using ResNet50 pretrained on Places 365.

If we keep every other parameter the same but we swap the pretrained weights we get these:

Left: ResNet50-ImageNet (we can see more animal features) Right: ResNet50-Places365 (human built stuff, etc.).

Dreaming is performed on multiple image resolutions stacked "vertically" (we call this an image pyramid).

Going from left to right the only parameter that changed was the pyramid size (from left to right: 3, 7, 9 levels).

Playing with pyramid ratio has a similar/related effect - the basic idea is that the relative area of the image which the deeper neurons can modify and "see" (the so-called receptive field of the net) is increasing and we get increasingly bigger features like eyes popping out (from left to right: 1.1, 1.5, 1.8):

Note: you can see the exact params used to create these images encoded into the filename!

Make sure to check out the Jupyter notebook!, I've explained this thoroughly.

Here are some further examples that you can create using this code!

The idea here is that whatever the network dreams just feed that back to it's input and apply a geometric transformation.

If we apply only central zoom we get this:

Applying central zoom and at the same time applying a 3 degree rotation per frame yields this:

Finally if we do a simple translation (5 px per frame top left to bottom right direction):

Hopefully these did not break your brain - it feels like web 1.0 early 2000s. Bear with me.

Instead of feeding the output back to input we just apply the algorithm per frame and apply some linear blending:

Linear blending just combines the current frame with the last one so as to reduce the flicker (here I used 0.85)

Note: all of the deepdream images/GIFs were produced by me, credits for original image artists are given bellow.

1. git clone https://github.com/gordicaleksa/pytorch-deepdream
2. Open Anaconda Prompt and navigate into project directory cd path_to_repo
3. Run conda env create from project directory (this will create a brand new conda environment).
4. Run activate pytorch-deepdream (for running scripts from your console or setup the interpreter in your IDE)

That's it! It should work out-of-the-box executing environment.yml file which deals with dependencies.

Note: If you wish to use video functions I have - you'll need ffmpeg in your system path.

PyTorch pip package will come bundled with some version of CUDA/cuDNN with it, but it is highly recommended that you install a system-wide CUDA beforehand, mostly because of the GPU drivers. I also recommend using Miniconda installer as a way to get conda on your system. Follow through points 1 and 2 of this setup and use the most up-to-date versions of Miniconda and CUDA/cuDNN for your system.

Just run jupyter notebook from you Anaconda console and it will open up a session in your default browser.
Open The Annotated DeepDream.ipynb and you're ready to play!

Note: if you get DLL load failed while importing win32api: The specified module could not be found
Just do pip uninstall pywin32 and then either pip install pywin32 or conda install pywin32 should fix it!

You just need to link the Python environment you created in the setup section.

Navigate to/activate your env if you're using Anaconda (and I hope you do) and you can use the commands I've linked below.

Tip: Place your images/videos inside the data/input/ directory and you can then just reference your files (images/videos) by their name instead of using absolute/relative paths.

To create some static Deep Dream images run the following command:

python deepdream.py --input <img_name> --img_width 600

This will use the default settings but you'll immediately get a meaningful result saved to:

data/out-images/VGG16_EXPERIMENTAL_IMAGENET/

Note: the output directory will change depending on the model and pretrained weights you use.

To get the out-of-the-box Ouroboros 30-frame video do the following:

python deepdream.py --input <img_name> --create_ouroboros --ouroboros_length 30

It will dump the intermediate frames to data/out-videos/VGG16_EXPERIMENTAL_IMAGENET/ and it will save the final video to data/out-videos.

To create a Deep Dream video run this command:

python deepdream.py --input <mp4 video name>

It will dump the intermediate frames to data/out-videos/tmp_out and it will save the final video to data/out-videos.

Well, enjoy playing with this project! Here are some additional, beautiful, results:

A GPU with 2+ GBs will be more than enough.

You'll be able to create DeepDream images, Ouroboros and DeepDream videos.

If you don't have a GPU, the code will automatically run on the CPU but somewhat slower (especially for videos).

If you're having difficulties understanding DeepDream I did an overview of the algorithm in this video:

And also the Jupyter Notebook I created is the best place to start!

I found these repos useful (while developing this one):

• deepdream (Caffe, original repo)
• DeepDreamAnim (Caffe)
• AI-Art (PyTorch)
• neural-dream (PyTorch)
• DeepDream (PyTorch)

I found the images I was using here:

• awesome figures pic
• awesome bridge pic

Other images are now already classics in the NST and DeepDream worlds.

Places 365 pretrained models came from this awesome repo.

If you find this code useful for your research, please cite the following:

@misc{Gordić2020DeepDream,
  author = {Gordić, Aleksa},
  title = {pytorch-deepdream},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/gordicaleksa/pytorch-deepdream}},
}

If you'd love to have some more AI-related content in your life 🤓, consider:

• Subscribing to my YouTube channel The AI Epiphany 🔔
• Follow me on LinkedIn and Twitter 💡
• Follow me on Medium 📚 ❤️