This repository provides an MSO attribute cleaned version of CelebA.

Haiyu Wu, Grace Bezold, Manuel Günther, Terrance Boult, Michael C. King, Kevin W. Bowyer, "Consistency and Accuracy of CelebA Attribute Values", CVPRW, 2023, arXiv:2210.07356

We report the first systematic analysis of the experimental foundations of facial attribute classification. Two annotators independently assigning attribute values shows that only 12 of 40 common attributes are assigned values with ≥ 95% consistency, and three (high cheekbones, pointed nose, oval face) have essentially random consistency. Of 5,068 duplicate face appearances in CelebA, attributes have contradicting values on from 10 to 860 of the 5,068 duplicates. Manual audit of a subset of CelebA estimates error rates as high as 40% for (no beard=false), even though the labeling consistency experiment indicates that no beard could be assigned with ≥ 95% consistency. Selecting the mouth slightly open (MSO) for deeper analysis, we estimate the error rate for (MSO=true) at about 20% and (MSO=false) at about 2%. A corrected version of the MSO attribute values enables learning a model that achieves higher accuracy than previously reported for MSO.

If you use any part of our code or data, please cite our paper.

@inproceedings{wu2023consistency,
  title={Consistency and accuracy of celeba attribute values},
  author={Wu, Haiyu and Bezold, Grace and G{\"u}nther, Manuel and Boult, Terrance and King, Michael C and Bowyer, Kevin W},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3257--3265},
  year={2023}
}

The statistic information is shown in this table. Note that, for label value, only MSO is cleaned in this paper.

You can download cleaned labels here.

Models are available in Model zoo

Download the cropped and aligned CelebA and run the script to get the dataset.

python drop_noise_ims.py -s /path/to/celeba/ims -f ./cleaned_mso_labels/partition.csv -d ./dataset

Download the weights from model zoo and run the script.

python test.py -m /path/to/weights -i ./dataset/test/ -l ./cleaned_mso_labels/label_cleaned_mso.csv