hi, @MingLin-home , In paper and official code, retrain the searched model by the feature loss cost too much resource, the 1.2ms latency model is very difficult to train completely end to end. so whether the series model‘s log can be released?

Dear researcher, Hello
I ran into some problems while running the code that I couldn't solve when I wanted to test CIFAR10
（python val_cifar.py --dataset cifar10 --gpu 0 --arch zennet_cifar10_model_size05M_res32）

（RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.）

The version I use is Torch1.7.1 Torchvision0.8.2 cudatoolkit10.2.89. May I ask if this is a problem with my operating environment? Or do I need to make changes in the code? Thanks a million, dear researcher.

Hi, everyone. I'm currently trying to reproduce your previous work "Neural Architecture Design for GPU-Efficient Networks", here is the repo link. After determining the model structure, such as "GENet_large", "GENet_small", etc. Can we refer to the training script(train_image_classification.py) in this repo to train the model and get the effect consistent with the paper description?

Hello, @MingLin-home ! I'm studing your zon_ NAS, but I encountered some problems in the process of re engraving. My results are as follows：

python val.py --fp16 --gpu 0 --arch zennet_imagenet1k_flops400M_SE_res224
Evaluate zennet_imagenet1k_flops400M_SE_res224 at 224x224 resolution.
/home/jet/anaconda3/lib/python3.9/site-packages/torchvision/transforms/transforms.py:287: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
warnings.warn(
---debug use_se in SuperResIDWE1K7(16,40,2,40,1)
---debug use_se in SuperResIDWE1K7(40,64,2,64,1)
---debug use_se in SuperResIDWE4K7(64,96,2,96,5)
---debug use_se in SuperResIDWE2K7(96,224,2,224,5)
loading pretrained parameters...
Using GPU 0.
/home/jet/anaconda3/lib/python3.9/site-packages/torch/nn/functional.py:3631: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
warnings.warn(
mini_batch 0, top-1 acc= 0%, top-5 acc= 0%, number of evaluated images=128
mini_batch 100, top-1 acc= 0%, top-5 acc= 0%, number of evaluated images=12928
mini_batch 200, top-1 acc= 0%, top-5 acc= 0%, number of evaluated images=25728
mini_batch 300, top-1 acc= 0%, top-5 acc=1.5625%, number of evaluated images=38528
*** arch=zennet_imagenet1k_flops400M_SE_res224, validation top-1 acc=0.08999999612569809%, top-5 acc=0.343999981880188%, number of evaluated images=50000

1. I hope you can help me to explain the meaning of these results.
2. I want to know : if I want to train a NAS, what procedures will I use?
   Thank you very much for your help!

As shown in Guideline 3 and the yellow underline.

计算方差以及得分的代码对应lightweight-neural-architecture-search/tinynas/scores中的那一个呢？计算特征图方差是通过模型前向推理数据得来的吗？还是将其转化到以计算通道的方式，和deepmad类似？

hi, lin, I am interesting your group work , do you do some expriments on object detect target ?
I want to some nas work on object detect , zen-nas are attractive at speed, do you have some work or idea about this?

Thank you for your great work.

I have tried zenscore and get great kendall tau in plain network.

I wonder if zenscore can apply to MobileNet-like Search Space?

Looking forward to your reply.

Hi, interesting work!

I had a couple of clarifying questions regarding the evolution search and the models provided.

For the pretrained models, those were trained on a V100 if I understood the paper correctly?

The inference speed charts on the T4 and Pixel 2 are only from exporting to TensorRT rather than actually training on the T4 and Pixel 2?

Your proposed Zen-NAS is a very efficient way to search for neural network structures. I read your article and GitHub code carefully, and did my own search on your code, but one thing I found is that the network structure searched almost always repeats more times the deeper the network is the network block, and the first few layers of the network block are repeated once, for example, I used your code to search the structure of MNas (the search space has been changed according to MNas), MNas0.35 optimal structure: 

SuperConvK3BNRELU(3,16,2,1)SuperResMnasV1K3(16,8,1,16,1)SuperResMnasV3K3(8,8,2,8,3)SuperResMnasV3K5(8,16,2,8,3) SuperResMnasV6K5(16,32,2,16,3)SuperResMnasV6K3(32,32,1,32,2)SuperResMnasV6K5(32,64,2,32,4)SuperResMnasV6K3(64,112,1,64,1) SuperConvK1BNRELU(112,1280,1,1)
but I searched with your architecture and the structure is as follows
SuperConvK3BNRELU(3,8,2,1)SuperResMnasV1K3(8,8,1,8,1)SuperResMnasV3K5(8,16,2,8,1)SuperResMnasV3K5(16,24,2,8,1)SuperResMnasV3K5(24,64,2,40,1)SuperResMnasV3K5(64,24,1,48,1)SuperResMnasV3K5(24,64,2,176,4)SuperResMnasV3K5(64,48,1,256,5)SuperConvK1BNRELU(48,2048,1, 1)
so the search out of the structure compared to the original structure is not very good, the structure is still the problem mentioned above, the search out of the network shallow block duplication for 1, only the deeper network has block duplication, also tested your code in the Flops400M,600M,900M, found the same problem, this is why?

Hi. I am interesting in what is the parameter setting for num param=0.5m

budget_model_size=5e5
max_layers=18
population_size=512
evolution_max_iter=480000  # we suggest evolution_max_iter=480000 for

Is it like this one?

Training for 480 epoch on single GPU will take quite a lot of time. So I wanted to check if multi GPU training is possible in the current code. I tried --dist_mode with mpi and auto are not supported as global_utils.AutoGPU() is not defined. But looks like --dist_mode=horovod is supported. Can it be used to do multi GPU training?
Is it possible to share sample command to do multi-gpu training?

Btw it took around 2.83 hrs to complete 1 epoch on my GTX 1080-TI, does this sound reasonable or too high? Typically I have seen on my machine (4 x GTX1080) around 15 mins per epoch for EfficientNet-ES using Ross wightman's timm repo.

Hello， Can i do Nas for myself data set based on your source code？

As I understand, the proposed Zen-score is an approximation of Phi-score for searching models with BN. If so, why don't you firstly use Phi-score to search an architecture without BN, then add BN layers to it?

My block look like this.

BottleneckCSP(
  (cv1): Conv(
    (conv): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act): Mish()
  )
  (cv2): Conv2d(128, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (cv3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
  (cv4): Conv(
    (conv): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (act): Mish()
  )
  (bn): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (act): Mish()
  (m): Sequential(
    (0): Bottleneck(
      (cv1): Conv(
        (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
      (cv2): Conv(
        (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
    )
    (1): Bottleneck(
      (cv1): Conv(
        (conv): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
      (cv2): Conv(
        (conv): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
        (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act): Mish()
      )
    )
  )
)

forward(self, x)

def forward(self, x):
        d = self.m(self.cv1(x))
        y1 = self.cv3(d)
        y2 = self.cv2(x)
        return self.cv4(self.act(self.bn(torch.cat((y1, y2), dim=1))))

Hi. I wonder if you have use attention blocks or transformer related architecture in your network because I did not see it in the code.

The paper says searching cost is 0.5 GPU day with NVIDIA V100 GPU, half precision (FP16), batch size 64.
We use supplied script to search network, and set evolution_max_iter=50000, using V100, FP16, batch 64, the whole process spends much time。
for example, using ./Zen_NAS_ImageNet_latency0.8ms..sh to search 0.8ms model, we take 29 hours.
loop_count=47000/50000, max_score=308.6, min_score=293.333, time=28.213h
loop_count=48000/50000, max_score=308.6, min_score=294.01, time=28.78h
loop_count=49000/50000, max_score=308.6, min_score=294.837, time=29.3295h

in paper, evolutionary iterations is 96000, but using 50000, we still cannot achieve 0.5 GPU day to finish search model.

I wonder is there some problems in code or else

Hi, @MingLin-home
I search the zen-nas-1.0M in CIFAR10 by your default scripts, the result is that, params is less than 1M, but flops is 360M which is about twice as much as in the paper(160M).
Is this normal? How can I reproduce the level in the paper? Thank you.😄

Hello,
When computing the Zen-Score it involves expectations over input x and parameters theta.
In ZeroShotProxy/compute_zen_score.py, the approximation of such expectations is controlled by argument repeat in compute_nas_score, which is the number of samples from (x, theta).
Although the default value of repeat is 32 in ZeroShotProxy/compute_zen_score.py, it is set to be 1 (hard-coded) when calling compute_nas_score in evolution_search.py

To reproduce your results, should I keep using repeat=1? I am not sure if it works to approximate the expectation with only one sample. But using larger size of sample increases the search budget.

i follow the paper "Neural Architecture Design for GPU-Efficient Networks", and see "We also use ResNet-152 as teacher network to get soft-labels. The soft-label loss and the true-label loss are weighted 1:1." at page 8
how to use soft-labels，teach genet. like knowledge distillation? have any code in this repo?

I was experimenting with RegNet like search space but with little different group_size. If I use my hand designed network (model_hand) I get 74.05 % top-1 accuracy with latency L. My modest objective is to learn a model (model_zennas) of same latency (L) but at least gives same accuracy as hand designed network. I have kept training schedule exactly same between model_hand and model_zennas but I got 1.4% less.
When I look at ZenScore of model_hand it is 112 where as for model_zennas it is 136.0. So it looks like it is not about model_hand not being part of search possibilities. But some how ZenScore assigns lesser score to higher accuracy model.

So my question is, if there is some tweak I can do to the way ZenScore is defined so that model_hand has similar score to model_zennas. If that happens I can expect model_zennas also generates similar accuracy as model_hand.

I like your way of searching much as it does not need training so exploring to accomplish above. Do you think it is appropriate for such objective?

I have a question. why the model's syncflow closer to 0 is better than other models whose absolute value of syncflow is much larger than 0?

From Figure 2. in the paper, it can be seen that deeper and wider network has higher Zen-Score. And Zen-Score positively correlates with model accuracy. So Deeper and wider network has higher accuracy, which is a well-known principle. Then whats' the meaning of Zen-Score?

First of all, thanks a lot for releasing the code for such a nice work. I have few doubts,

1. The seach_space_block_type_list_list in both files, search_space_IDW_fixfc.py as well as search_space_XXBL.py are same. Is it intended?

2. The script, Zen_NAS_ImageNet_flops400M.sh and Zen_NAS_ImageNet_latency0.2ms use same search space. But the paper mentions Zen_NAS_ImageNet_flops400M.sh using MB block and Zen_NAS_ImageNet_latency0.2ms uses botn block like Resnet50.

Thanks again.

The link you provided for zennet_imagenet1k_flops400M_SE_res224 model - https://idstcv.oss-cn-zhangjiakou.aliyuncs.com/les/iccv2021_zennet_imagenet1k_flops400M_SE_res224/student_best-params_rank0.pth is not working.

Please update the link in the readme

hello, nice work. Do you have any plan to release search code?

Train ZenNet-2.0M in CIFAR100

Test model： zennet_cifar100_model_size2M_res32

Data augmentation：

• subtracting the channel mean and dividing the channel standard deviation
• mixup
• label-smoothing
• random erasing
• random crop/resize/flip/lightting
• Auto Augment

Train optimizer：

• SGD optimizer with momentum 0.9
• weight decay 5e-4 for CIFAR10/100
• Learning rate 0.1 with batch size 256
• Cosine learning rate decay
• 1440 epochs in CIFAR10/100

Hi, your work is exciting and inspires us a lot. So we try to reproduce，but we train the test model according to the above configuration, and the training top-1 accuracy is only 80% in cifar100. Did we make a mistake or miss something?

Dear researcher,
There was a file missing from the code (analyze_model.py) when I finished training the model. Always feel this module is very important. Could you help me find out what the problem is? Thank you very much

I'm new to NAS and I wonder how can I do searching in HardNet.

Hello,

As I remember from the paper, your method works on Vanilla CNN. However, in algorithm 1, you just mentioned that residual connections are deleted.

It confused me a little bit, and I do not know your method can be applied on any CNN without residual connection or it only can be applied on Vanilla CNN.

Can I use your code on benchmarks such as NAS-Bench-201?

I want to use ZenNAS on a medical image dataset, but am confused when you say ZenNAS is "data-free" and "a data independent method. " I understand how it is "training-free", and that the calculation of Zen-Score only takes a few forward inferences through a randomly initialized network, but is this zero-shot Zen-Score based at all on my data?

Hello MingLin,

I found your NAS approach to be very interesting.

Did you ever try to combine the ZEN-Score with the NTK-score like the TE-NAS paper did? The TE-NAS paper suggests that combining the NTK-Score with a expressivity measure would improve the overall performance.

Thanks,
SXK