Neural architecture search

On this page 7

Neural architecture search (NAS) is the automated process of designing the structure of a neural network by searching a defined space of candidate architectures and selecting those that perform best on a target task. Within neural networks and artificial intelligence more broadly, it treats choices such as layer types, layer counts, connection patterns and widths as variables to be optimized rather than decisions made by hand.

NAS emerged as a research field around 2016 and produced several widely used model families, including NASNet, EfficientNet and MobileNetV3. Its central trade-off is between search quality and compute cost: early methods consumed thousands of GPU days, while modern techniques reduce the cost to a few GPU days or less at the price of stronger assumptions.

Every neural architecture search method is defined by three interacting components: a search space, a search strategy and a performance estimation strategy. Changing any one of them changes what the method can discover and how much it costs.

The table below describes each component and its main design decisions.

ComponentWhat it definesTypical design choices
Search spaceWhich architectures are possibleLayer operations, connections, depth, width
Search strategyHow candidates are proposedReinforcement learning, evolution, gradients
Performance estimationHow candidates are scoredFull training, proxies, weight sharing
Building blocks of a NAS method

The search space encodes prior knowledge and therefore limits results: a space built only from convolutional layers cannot produce a Transformer. A popular design is the cell-based search space, in which the method searches for one small repeating block, called a cell, that is then stacked to form the full network. This reduces the problem size dramatically and lets a cell found on a small dataset transfer to a larger one.

Search strategies and how they explore

A search strategy determines how new candidate architectures are generated from the results of earlier ones. The main families differ in sample efficiency, parallelism and the assumptions they make about the search space.

The comparison below summarizes the established strategy families.

StrategyHow it worksStrengthWeakness
Reinforcement learningA controller network samples architectures and learns from rewardsFlexible, discrete spacesVery high sample count
Evolutionary searchMutates and selects a population of architecturesRobust, easy to parallelizeMany full evaluations
Gradient-based searchRelaxes choices into continuous weights, as in DARTSFast, often under one GPU dayUnstable, may favor skip connections
Bayesian optimizationModels expected performance and picks promising candidatesSample-efficientHard to scale to large spaces
Random searchSamples architectures uniformlyStrong, honest baselineNo learning between samples
Main NAS search strategies

Research has repeatedly shown that well-tuned random search within a good search space performs surprisingly close to sophisticated strategies. This finding indicates that much of the success attributed to NAS comes from the design of the search space itself, a point experienced practitioners treat as a key lesson.

Making performance estimation affordable

Performance estimation is the most expensive part of neural architecture search, because the honest way to score a candidate is to train it to convergence and measure validation accuracy. The original reinforcement learning NAS of 2016 to 2017 used hundreds of GPUs for weeks, which motivated a family of shortcuts.

The following table lists the main cost-reduction techniques and the accuracy risk each introduces.

TechniqueIdeaCost reductionMain risk
Low-fidelity proxiesTrain fewer epochs or on smaller dataModerateRankings change at full scale
Learning curve extrapolationPredict final accuracy from early epochsModeratePoor predictions for unusual curves
Weight sharing (one-shot)All candidates share weights of one supernetVery largeShared weights distort rankings
Zero-cost proxiesScore untrained networks from gradients or activationsExtremeWeak correlation on many tasks
Performance predictorsSurrogate model trained on evaluated architecturesLargeNeeds a seed set of results
Performance estimation techniques

Weight sharing made NAS practical for most organizations. A single supernet contains every candidate as a subgraph; training it once allows any subnetwork to be evaluated without separate training. The cost is a ranking correlation problem: an architecture that looks strong with shared weights may not remain strong when trained alone.

Hardware-aware search and practical use

Modern neural architecture search usually optimizes several objectives at once, most often accuracy together with latency, memory or energy on a specific device. This hardware-aware NAS adds measured or predicted inference latency to the reward, so the search returns a Pareto front of models rather than a single winner.

This approach produced MobileNetV3 and EfficientNet, whose architectures were tuned for mobile processors and then scaled with fixed rules. The Once-for-All method extends the idea by training one large network from which subnetworks for many devices can be extracted without retraining.

NAS is most useful when an organization deploys one model to many devices or millions of inferences, where small efficiency gains repay the search cost. For a single project on a common task, a proven published architecture combined with careful hyperparameter tuning is usually the more economical choice.

Limitations and common misconceptions

Neural architecture search automates part of model design but does not remove human expertise, because people still define the search space, the objectives and the training recipe. Several limitations are widely recognized in the field.

  • Unfair comparisons: reported gains often mix the effect of the architecture with better training recipes or data augmentation.
  • Reproducibility gaps: results vary with random seeds, and benchmarks such as NAS-Bench-101 and NAS-Bench-201 were created partly to address this.
  • Narrow novelty: NAS optimizes within a known design vocabulary and has rarely invented fundamentally new building blocks such as attention.
  • Limited impact on very large models: at the scale of large language models, a single training run is too costly for broad search, so searches focus on small proxies and scaling rules.
  • Overfitting to the search dataset: architectures tuned heavily on one benchmark may transfer less well than expected.

NAS is best understood as a specialized form of hyperparameter optimization for network topology, closely related to AutoML. Treating it as a fully autonomous designer leads to unrealistic expectations about what the search can deliver.

The short version

Designing a neural network involves many choices: how many layers, what kind, and how they connect. Neural architecture search lets a computer try many designs automatically and keep the ones that work best, much as a breeder selects the strongest plants over generations.

The trade-off is effort. Testing each design properly takes considerable computing power, so most modern methods use clever shortcuts to estimate quality quickly. People still decide which designs are allowed and what counts as good.

Where it makes a difference

A company building a smartphone camera app needs an image segmentation model that separates people from the background in real time. The requirement is under 15 milliseconds per frame on a mid-range phone processor while matching the accuracy of a larger server model.

The machine learning team defines a search space of mobile-friendly blocks with varying kernel sizes, expansion ratios and depths, trains one weight-sharing supernet over about four days on eight GPUs, and runs an evolutionary search that scores candidates with a latency lookup table measured on the target phone. After retraining the best three candidates from scratch, the team selects one at 12 milliseconds per frame.

The result is a model that is about 30 percent faster than the hand-picked baseline at equal accuracy, which allows the feature to run on older phones that would otherwise have been excluded.