Describe the Question A clear and concise description of the question.

• topology search space in NATS-Bench

Hi, I have some questions about how to use val and test acc when searching on cifar10 subset.

First, when training an arch search algo, we use val acc as labels during the training procedure. And to evaluate the algo, the searched arch's test acc is queried as result. This works for cifar100 and imagenet16 subset since they offer val and test acc(I really don't know what valtest_acc for)

For cifar10, there are two subsets: cifar10-val and cifar10. The former offers val and test acc, so the above train and eval pipeline still works,

but the latter only offers test acc, so when using the latter subset, I can not build labels since there are no val acc.

Above information is offered by api.get_more_info

Many thanks

FLOPS counts

• about the topology search space in NATS-Bench?

Hi, the question is that when I using the flops counts tools to calculate the flops of the model in nasbench201, the results are always larger than the flops offered by 201 dataset, for example arch index 0, cifar10-valid flops 15.64737, cifar100 15.65322, and imagenet16 flops 3.91948, however from the thop tools they are 16.464576, 16.470336, 4.123712, from the ptflops tools they are 16.810634, 16.816484, 4.210296. But either 201 dataset or the above tools offer the same params counts.

are there any operations that the model does not count?

Many thanks for your excellent works and code！

Hi, I want to use the NATS-Bench, but I still have some questions about the results of benchmark algorithms given in the paper. (1) Hp=200 in tss and hp=90 in sss, right? (2)the results is top1 or top5 or others? (3)Sorry, but can I trouble you to wirte a usage example to show how can I get the correct metrics when I search a new arch? Thanks!

Hi, I would like to use you NATS-Bench for other datasets except cifar and Imagenet, and with higher resolutions like (256*256). Is it possible to sample a network like what you did for cifar below and then change the cells resolutions?

import xautodl, nats_bench

from nats_bench import create
from xautodl.models import get_cell_based_tiny_net

api = create(None, 'tss', fast_mode=True, verbose=True)

config = api.get_net_config(12, 'cifar10')
network = get_cell_based_tiny_net(config)

#then a code to change the input resolution to the target size of 256*256

Thanks for your response

In the README.MD, there is only one example to generate an architecture at index 12 with get_cell_based_tiny_net.

But in the codebase, I found there are many functions including:

1. get_cell_based_tiny_net
2. obtain_model

Which is the real model at index 12? (The one whose performance is recorded in get_more_info).

If I want to obtain the model/architecture at the index 12 whose performance is the exactly measured and recorded in get_more_info. Which method should i use ?

Thank you

Hi! I want to get the validation and test accuracy as Table 4 in "NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size" paper. I just want to check the following commands are correct or not:

After finishing the architecture search (I'm studying Weight-Sharing approach), I get the genotype. Then, I get the arch_index via: arch_index = api.query_index_by_arch('......genotype here......') Therefore, for Cifar10 validation accuracy: info = api.get_more_info(arch_index, 'cifar10-valid', hp=200) for Cifar10 test accuracy: info = api.get_more_info(arch_index, 'cifar10', hp=200) for Cifar100 validation/test accuracy: info = api.get_more_info(arch_index, 'cifar100', hp=200) for ImageNet16-120 validation/test accuracy: info = api.get_more_info(arch_index, 'ImageNet16-120', hp=200)

Following are some points I want to check:

1. Does Cifar10 test accuracy results use train + valid set for training and test set for testing? Thus, I should use 'cifar10' instead of 'cifar10-valid' to get test accuracy?
2. What does valtest-accuracy mean in Cifar100 and ImageNet16-120?
3. I get the architecture with validation accuracy higher than Optimal values reported in the paper for ImageNet16-120. Why?

Great Thanks!

Describe the bug For some architectures, get_cost_info() returns different flops/params for hp="12" and hp="200" in topology search space. For example, the number of parameters (for CIFAR-10) of architecture |skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2| is 0.802426 for hp="12", but for hp="200", the value is 0.6403993333333333.

I've found many other architectures in which the same behavior occurs, but some specific examples are as follows:

• |nor_conv_1x1~0|+|nor_conv_3x3~0|skip_connect~1|+|none~0|skip_connect~1|none~2|
• |nor_conv_3x3~0|+|skip_connect~0|skip_connect~1|+|none~0|skip_connect~1|none~2|

To Reproduce

from nats_bench import create

arch = "|skip_connect~0|+|none~0|nor_conv_3x3~1|+|avg_pool_3x3~0|nor_conv_3x3~1|nor_conv_3x3~2|"
api_path = "NATS-tss-v1_0-3ffb9-simple"
search_space = "tss"
dataset = "cifar10"

api = create(api_path, search_space, fast_mode=True, verbose=False)
idx = api.query_index_by_arch(arch)
info_12 = api.get_cost_info(idx, dataset, hp="12")
info_200 = api.get_cost_info(idx, dataset, hp="200")
print(info_12)
print(info_200)

{'flops': 113.95137, 'params': 0.802426, 'latency': 0.016719988563604522, 'T-train@epoch': 21.51544686158498, 'T-train@total': 258.1853623390198, 'T-ori-test@epoch': 1.496117415882292, 'T-ori-test@total': 17.953408990587505}
{'flops': 90.35841, 'params': 0.6403993333333333, 'latency': 0.016719988563604522, 'T-train@epoch': 21.515446861584987, 'T-train@total': 4303.0893723169975, 'T-ori-test@epoch': 1.4961174158822923, 'T-ori-test@total': 299.22348317645844}

• OS: Ubuntu 20.04.3 LTS (Focal Fossa)
• Python version: 3.8.12
• PyTorch version: 1.8.1+cu102

Expected behavior I think flops/params for hp="12" and hp="200" should be the same. (From reading the paper and your implementation in AutoDL-Projects, I could not find any factors that would make a difference.)

Describe the Question A clear and concise description of the question.

• Is it about the topology search space in NATS-Bench? No
• Is it about the size search space in NATS-Bench? Yes
• Which figure or table are you referring to in the paper? N/A

I would like to generate a computational graph from a sample from the size search space. Do you provide any functionality for that? I presume that you must have used this functionality when creating this dataset, so if it is not available within the NATS-Bench codebase, can you give some pointers as to how I can do this?

Thanks!

I have downloaded archive(tss) file and used tar command then uploaded the folder ungenerated on google drive to access it from colab for below code: from nats_bench import create api = create('/content/drive/MyDrive/NATS-tss-v1_0-3ffb9-simple/', 'tss', fast_mode=True, verbose=True)

I get: FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/NATS-tss-v1_0-3ffb9-simple//meta.pickle.pbz2.pbz2' why is that?

Which Algorithm Size Search Space Cifar 10

Describe the Question I get all 32768 architecture candidates‘ acc using the code followed, but all of them in under 90%. The results in your paper of all method is above 90%. How can I get the result of table 4 in your paper?

from nats_bench import create model_cifar10_rank = {} api = create(None, 'sss') for index in range(32768): info = api.get_more_info(index, 'cifar10') config = api.get_net_config(index, 'cifar10') model_cifar10_rank[config['channels']] = info['test-accuracy']

models_cifar10_acc_rank 2.txt

I downloaded the archive with the evaluations of NATS Bench and I would like to get more information about the format of the data. In particular, the data linked in this page https://pypi.org/project/nats-bench/ at the 'archive' links.

I already have software for running my experiments, I would just like to convert the data format instead of integrating a new library, which may be more suitable for a different setting.

Of course, I will cite the paper as suggested.

Hello. I would like to know if the checkpoints split are independent to each others. I meant if i download one of those 30Gb split does it contains all the information of a set of specific architectures or do i need to download others checkpoints and jointly unzip them in order to be able to use them.

Where can i find the transform classes for inference if i want to evaluate the retrieved checkpoint to reproduce same test accuracy in the model information? thank you?

Describe the bug When retrieving network parameters with get_net_param I obtain an empty dict that I clearly cannot import as state_dict of a Pytorch model.

To Reproduce Please provide a small script to reproduce the behavior:

from nats_bench import create
nats_api = create("NATS-tss-v1_0-3ffb9-simple", search_space="topology", fast_mode=True, verbose=False)
# sample random idx to simulate behavior
random_idx = nats_api.random()
# get corresponding architecture on CIFAR10
random_architecture = nats_api.get_net_config(index=random_idx, dataset="cifar10")
# retrieve network parameters according to main README.md
random_params = nats_api.get_net_param(index=random_idx, dataset="cifar10", seed=None)
print(random_params)
# prints {111: None}
print(next(iter(random_params.values())))
# prints None

OS: macOS Ventura 13.0.1 Python version: 3.10.8 Pytorch version: 1.13.0 Expected behavior random_params should be a meaningful dictionary that I could use as params dict for a torch model

I was working with NAS_201 bench earlier and now am shifting to NATS bench. I have the following doubts:

• Will the index of architecture obtained with NAS_201 and NATS bench same for a given arch? Because when using both NAS_201 is giving 12804 index while NATS bench is returning -1. Also the index of NAS_201 provides different config arch when used as index in NATS.
• While obtaining weights I am getting an empty return when using the NAS_201 index.

Following is my implementation. I am using benchmark file with sss.

api = create(d, 'sss', fast_mode=False, verbose=True) index = api.query_index_by_arch(convert_naslib_to_str(best_arch)) config = api.get_net_config(index, 'cifar10') best_arch = get_cell_based_tiny_net(config) logger.info("Queried results ({}): {}".format(metric, best_arch)) params = api.get_net_param(index, 'cifar10', None) best_arch.load_state_dict(next(iter(params.values())))

I am not sure if there is val/test part in ImageNet16120 dataset. I only found files of train set and val set in document you provided. But in Readme document, the test set mentioned is actually val set, according to your source code.

If the val set is used for test or just wrongly named, did you use the whole training set for model training without spliting?

Hello, first of all, thank you very much for your work in the  field of NAS. Here are my questions about NATS-Bench，and hope to get your confirmation：

1. What are the specific units for results such as 'train-all-time'? ( for example, minutes?)
2. What does training cost mean specifically? In particular, what is the difference between 'train-all-time' and 'T-train@total'?