Quan Minh Phan, Ngoc Hoang Luong

In GECCO 2024.

• Clone this repo
• Install necessary packages and databases.

$ cd MF-NAS
$ bash install.sh

This repo have already implemented following NAS algorithms:

• Random Search (RS)
• Local Search: First-improvement Local Search (FLS); Best-improvement Local Search (BLS)
• Successive Halving (SH)
• Regularized Evolution (REA)
• REA + Warmup (REA+W)
• MF-NAS (Ours)

Our experiments are conducted on NAS-Bench-101, NAS-Bench-201, and NAS-Bench-ASR search spaces.

The configurations of algorithms are set in configs/algo_101.yaml, configs/algo_201.yaml, and configs/algo_asr.yaml. The configurations of problems are set in configs/problem.yaml.

To reproduce all main results in our paper, run the below scripts:

$ python /script/run_101.sh
$ python /script/run_201.sh
$ python /script/run_asr.sh

To reproduce the ablation studies, run the below scripts:

$ python /script/run_201_with_different_zc_metrics.sh

$ python /script/compare_RS_and_LS.sh

$ python /script/replace_val_acc_with_train_loss.sh

Note that you can search with other metrics. However, the using_zc_metric and metric hyperparameters must be set so that they do not conflict with each other.

For example, if you use the synflow/jacov metrics as the search objective, you need to set using_zc_metric to True. If you use val_acc/train_loss as the search objective, you must set using_zc_metric to False.

The table of performance metrics that are currently available for all networks in each search space.

All metrics are logged as .pickle and .json files.

Here are our best results (performance & search cost) for each search space:

We want to give our thanks to the authors of NAS-Bench-101, NAS-Bench-201, and NAS-Bench-ASR for their search spaces; to the authors of Zero-cost Lightweight NAS and NAS-Bench-Zero-Suite for their zero-cost metric databases.