---

Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition [arxiv] - NeurIPS 2023 (Oral)

Table of contents

• Setup
• Download datasets
• Download raw data files
• Create Configs
• Taining and evaluation
• Joint NAS+HPO
  • Search
  • Training
  • Analysis
• Evaluation with pre-trained models

Setup

To setup your environment use the commands below:

git clone https://github.com/dooleys/FR-NAS/
cd FR-NAS
conda create --name frnas python=3.9.11
conda activate frnas
pip install -r requirements.txt

Download datasets

Create configs

Dataset	Download link	Split
CelebA	download	Train-Val-Test
RFW	download	Test
VGGFace2	train-test	Train-Test
AgeDB	download	Test
LFW	download	Test
CFP_FF	download	Test
CFP_FP	download	Test
CALFW	download	Test
CPLPW	download	Test

Note: Folder splits/ contains the identities used in the train-val-test split for CelebA and VGGFace2. Please use these files to create the necessary splits from the downloaded data.

If you want to compute disparities across agegroups on the agedb dataset prepare the dataset following instructions here

Large-scale study of fairness of architectures

The set of experiments below are for the large-scale analysis we conduct for architectures and their biases. This study is a driving force to motivate the use of NAS and HPO in the next experiments.

Modify user configs

After downloading the datasets above change the dataset and metadata paths in user_configs/config_user_celeba.yaml and user_configs/config_user_vgg.yaml to their respective locations. If you want to use comet for logging make sure you set the appropriate username and api key in these config files.

Create configs

bash scripts/create_configs_celeba.sh
bash scripts/create_configs_vgg.sh

Train and evaluate architectures

CelebA

bash scripts/experiments_default_celeba.sh
bash scripts/experiments_multi_celeba.sh

VGGFace2

bash scripts/experiments_default_vgg.sh
bash scripts/experiments_multi_vgg.sh

Analysis

Joint NAS+HPO

After the large-scale analysis we design a search space inspired by the Dual-Path-Network (DPN) and perform joint NAS and HPO using bayesian optimization in the SMAC framework.

Before running the search make sure to change the dataset paths and the user configs in search_configs/config_celeba.yaml and search_configs/config_vggface.yaml. Then follow the following steps sequentially

Search

Launch the scheduler

sbatch scripts/scheduler_dpn.sh

Initialize the workers

Note: Launch the workers after the scheduler has started. Launch as many workers as you want by submitting the script below multiple times.

sbatch scripts/workers_dpn.sh

Start the distributed search

Note: You need to start schedulers and workers separately for search on CelebA and VGGFace2

sbatch scripts/job_celeba.sh
sbatch scripts/job_vgg.sh

Other NAS methods

Search

To run other NAS methods like NSGA-II and MO-ASHA (on VGGFace2) use the following commands

python src/search/search_nsga.py
python src/search/search_moasha.py

Train from scratch

To run other NAS methods like NSGA-II and MO-ASHA (on VGGFace2) use the following commands

python src/search/train_synetune_archs.py --config moasha
python src/search/train_synetune_archs.py --config nsga2

Training

Train architectures discovered on CelebA

python src/search/train_smac_arch_celeba.py --seed 111 --config config1
python src/search/train_smac_arch_celeba.py --seed 111 --config config2

Train architecture discovered on VGGFace2

python src/search/train_smac_arch_vgg.py --seed 111 

Analysis

After all the training and testing scripts are finished, use the analysis/analysis.py script to analyze the performance of the models. The anlysis centers on assembling a list of all the file paths into a list and passing that and the associated metadata to functions which perform the analysis and aggregate them into a dataframe. For example, if all your model output files live in pickle files in the subdirectories of the output/ folder, obtain the accuracy, error, disparity, rank disparity, ratio, rank ratio, and error ratio dataframes here:

files = glob.glob('output/**/*.pkl')
metadata = pd.read_csv(f'vggface/vggface2_val_identities_gender.csv')

acc_df_vgg, acc_disp_df_vgg, rank_df_vgg = analyze_pickle_files(files, metadata)
_, acc_disp_ratio_df_vgg, rank_ratio_df_vgg = analyze_pickle_files(files, metadata, ratio=True)
err_df, error_ratio_df, _ = analyze_pickle_files(files, metadata, ratio=True, error=True)                

The results for our training and testing runs can be found in the csvs in the analysis/ folder. There is also the notebook Paper_plots.ipynb which reproduces the main figures in the paper.

Comparison with Fairness Baselines

Angular

python src/baselines/smac_dpn_trainer_angular_celeba.py --seed 111 
python src/baselines/smac_dpn_trainer_angular_vgg.py --seed 111 

Label Flip

python src/baselines/smac_dpn_trainer_flip_celeba.py --seed 111 
python src/baselines/smac_dpn_trainer_flip_vgg.py --seed 111 

SensitiveNets

python src/baselines/smac_dpn_trainer_discriminator_celeba.py --seed 111 
python src/baselines/smac_dpn_trainer_discriminator_vgg.py --seed 111 

Evaluation on FR Benchmarks

You can download our pretrained models from here and save it it your current directory. To evaluate the models on different face recognition benchmarks:

VGGFace2

python src/evaluation/eval_fr.py --backbone smac_301 --optimizer SGD --dataset vggface2 --head CosFace
python src/evaluation/eval_fr.py --backbone dpn107  --optimizer SGD --dataset vggface2 --head CosFace
python src/evaluation/eval_fr.py --backbone dpn107  --optimizer AdamW --dataset vggface2 --head CosFace
python src/evaluation/eval_fr.py --backbone rexnet_200 --optimizer SGD --dataset vggface2 --head CosFace

CelebA

python src/evaluation/eval_fr.py --backbone smac_000 --optimizer SGD --dataset celeba --head CosFace
python src/evaluation/eval_fr.py --backbone smac_010 --optimizer SGD --dataset celeba --head CosFace
python src/evaluation/eval_fr.py --backbone smac_680 --optimizer SGD --dataset celeba --head CosFace
python src/evaluation/eval_fr.py --backbone dpn107 --optimizer SGD --dataset celeba --head CosFace
python src/evaluation/eval_fr.py --backbone dpn107 --optimizer SGD --dataset celeba --head MagFace

Bibtex

If this repo is helpful to you, please consider to cite it. Thank you! :)

@inproceedings{fair-nas-2023,
  title={Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition},
  author={Dooley, Samuel and Sukthanker, Rhea and Dickerson, John P and White, Colin and Hutter, Frank and Goldblum, Micah},
  booktitle={Advances in Neural Information Processing Systems},
  year={2023},
} 

Acknowledgements

Our training and evaluation pipelines are inspired from face.evoLVe