While image recognition and generation have been developed for several years and gotten great breakthrough in many areas, facial expression related models have been less touched due to the data complexity and limited benchmark datasets. Based on a novel database RAF-DB, we applied a popular discriminative model, the VGG-net to recognize the expressions so as to see the discrimination of different expressions. We then learned a low dimensional representation for each facial expression and developed a conditional architecture for generative adversarial networks. Our generative model can learn the conditional distribution of the expressions by taking the representations as the input and successfully generate fake images for each condition.

• NumPy
• SciPy
• Matplotlib
• Keras
• Tensorflow

For this project we applied the RAF-DB. To download the dataset, please follow the link http://whdeng.cn/RAF/model1.html and contact the group of reference 1. After downloading the dataset, move the folder "basic/" inside the folder "Data/" in our repository to run our code.

We choose VGG16 as our discriminative model. To instanstiate the VGG16 architecture. The input data is in folder /images/aligned and resize them to 160*160 as input data

To provide a pre-trained VGG16 model, VGG16_MODEL=tf.keras.applications.VGG16(input_shape=IMG_SHAPE,include_top=False,weights='imagenet')

Parameters:

1. include_top: whether to include the 3 fully-connected layers at the top of the network.
   
2. weights: one of None (random initialization), 'imagenet' (pre-training on ImageNet)
   
3. pooling: Optional pooling mode for feature extraction
   

Compile the model and train it

model.compile(optimizer=tf.train.AdamOptimizer(), 
              loss=tf.keras.losses.sparse_categorical_crossentropy,
              metrics=["accuracy"])
history = model.fit(train_ds,
                    epochs=100, 
                    steps_per_epoch=2,
                    validation_steps=2,
                    validation_data=validation_ds)

Parameters:

1. optimizer: String (name of optimizer) or optimizer instance
   
2. loss: String (name of objective function) or objective function or Loss instance
   
3. metrics: List of metrics to be evaluated by the model during training and testing
   

Go to the "Data/" folder. Open and follow the "Data_PreProcess.ipynb" file for data process. Open and follow the "Cov_AutoEncoder.ipynb" file to train the convolutional Autoencoder and applied PCA to get the expression represenations.

Go to the "Generative_model" foler. To train the cWGAN, run

python conditional_WGAN.py

After training process (have already recorded some checkpoints), to generate samples for the certain expression, run

python Expression_generator.py  <checkpoint path>  <expression>  <sample amount> <sample path>

• checkpoint path: the relative path of the check point files on which you want to generate samples.
• expression: "Surprise", "Fear", "Disgust", "Happiness", "Sadness", "Anger" or "Neutral".
• sample amonut: the number of samples you want generate.
• sample path: the path of the folder where you want to record the samples.

Examples of generated samples:

[1] Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild

[2] Very Deep Convolutional Networks for Large-Scale Image Recognition

[3] Generative Adversarial Nets

[4] Improved Training of Wasserstein GANs

[5] Conditional Generative Adversarial Nets

[6] De Novo Protein Design for Novel Folds using Guided Conditional Wasserstein Generative Adversarial Networks (gcWGAN)