Sketch inversion using DCNNs (synthesising photo-realistic images from pencil sketches) following the work of Convolutional Sketch Inversion and Scribbler.

This project is mainly focused on sketches of human faces and architectural drawings of buildings. However according to Scribbler and the followed experimentation with their proposed framework, it is possible that given a large dataset and ample training time, this network could generalize to other categories as well.

The model has been trained using a dataset generated from a large database of face images, and then the network was fine-tuned to fit the purpose of architectural sketches.

• Large-scale CelebFaces Attributes (CelebA) dataset.
• ZuBuD Image Database.
• CUHK Face Sketch (CUFS) database
• Building sketches from Google Images for testing

The datasets were simulated, i.e. the sketches were generated, using the following methods (with the exception of the CUHK dataset, which contains sketches and the corresponding images)

• Pencil Sketchify
• XDoG (Extended Difference of Gaussians)
• Neural Style Transfer

Furthermore, due to the low number of images of buildings available, various augmentations on the ZuBuD dataset were applied to produce more images using the following image augmentation tool

The network architecture used in Scribbler was used. The generator follows an encoder-decoder design, with down-sampling steps, followed by residual layers, followed by up-sampling steps.

The Loss function was computed as the weighted average of three loss functions; namely: pixel loss, total variation loss, and feature loss.

The pixel loss was computed as:

The feature loss was computed as:

The total variation loss was used to encourage smoothness of the output, and was computed as

Where phi(x) is the output of the fourth layer in a pre-trained model (VGG16 relu_2_2) to feature transform the targets and predictions.

The total loss is then computed as

L_t = w_pL_p + w_fL_f + w_vL_v

For the present application, w_f = 0.001, w_p = 1, w_v = 0.00001

• Face Weights after training the network on the CelebA dataset using the Pencil Sketchify method
• Building Weights after fine-tuning the network for the building sketches using the augmented ZuBuD dataset with the Pencil Sketchify method

• Training with a larger building dataset using a variety of sketch styles to improve the generality of the network
• Adding adversarial loss to the network.
• Using sketch anti-roughing to unify the styles of the training and input sketches.
• Passing the sketch results to a super-resolution network to improve image clarity.
• Increasing the image size of the training data.

Sketchback