White-Balance Emulator for Color Augmentation

What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance

Mahmoud Afifi¹ and Michael S. Brown<sup>1,2

1</sup>York University    ²Samsung AI Center (SAIC) - Toronto

Project page - Paper - Supplementary Materials

Our augmentation method can accurately emulate realistic color constancy degradation. Existing color augmentation methods often generate unrealistic colors which rarely happen in reality (e.g., green skin or purple grass). More importantly, the visual appearance of existing color augmentation techniques does not well represent the color casts produced by incorrect WB applied onboard cameras, as shown below.

Quick start

1. Python:

1. Requirements: numpy & opencv-python

• pip install numpy
• pip install opencv-python

2. Run wbAug.py; examples:

• Process a singe image (generate ten new images and a copy of the given image):
  • python wbAug.py --input_image_filename ../images/image1.jpg
• Process all images in a directory (for each image, generate ten images and copies of original images):
  • python wbAug.py --input_image_dir ../images
• Process all images in a directory (for each image, generate five images without original images):
  • python wbAug.py --input_image_dir ../images --out_dir ../results --out_number 5 --write_original 0
• Augment all training images and generate corresponding ground truth files (generate three images and copies of original images):
  • python wbAug.py --input_image_dir ../example/training_set --ground_truth_dir ../example/ground_truth --ground_truth_ext .png --out_dir ../new_training_set --out_ground_truth ../new_ground_truth --out_number 3 --write_original 1

3. demo.py shows an example of how to use the WBEmulator module

2. Matlab:

1. Run install_.m
2. Try our demos:
   • demo_single_image to process signle image
   • demo_batch to process an image directory
   • demo_WB_color_augmentation to process an image directory and repeating the corresponding ground truth files for our generated images
   • demo_GUI (located in GUI directory) for a GUI interface
3. To use the WB augmenter inside your code, please follow the following steps:
   • Either run install_() or addpath to code/model directories:

    addpath('src');
    addpath('models'); 
    %or use install_()
   

• Load our model:

  load('synthWBmodel.mat'); %load WB_emulator CPU model --  use load('synthWBmodel_GPU.mat');  to load WB_emulator GPU model
  

• Run the WB emulator:

  out = WB_emulator.generate_wb_srgb(I, NumOfImgs); %I: input image tensor & NumOfImgs (optional): numbre of images to generate [<=10]
  

• Use the generated images:

  new_img = out(:,:,:,i); %access the ith generated image
  

Dataset

We used images from Set1 of the Rendered WB dataset to build our method.

In our paper, we introduced a new testing set that contains CIFAR-10 classes to evaluate trained models with different settings. This testing set contains 15,098 rendered images that reflect real in-camera WB settings. Our testing set is divided into ten directories, each includes testing images for one of CIFAR-10 classes. You can download our testing set from the following links: 32x32 pixels | 224x224 pixels | 227x227 pixels

MIT License

Publication

If you use this code or our dataset, please cite our paper:

Mahmoud Afifi and Michael S. Brown. What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance. International Conference on Computer Vision (ICCV), 2019.

@InProceedings{Afifi_2019_ICCV,
author = {Afifi, Mahmoud and Brown, Michael S.},
title = {What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}

Related Research Projects

• When Color Constancy Goes Wrong: White balance camera-rendered sRGB images (CVPR 2019).
• Color Temperature Tuning: A camera pipeline that allows accurate post-capture white balance editing (CIC best paper award, 2019).
• Deep White-Balance Editing: A multi-task deep learning model for post-capture white-balance editing (CVPR 2020).