Hidden layer

On this page 7

A hidden layer is any layer of neurons that sits between the input layer and the output layer of a neural network and whose values are not directly observed in the training data. In neural networks, hidden layers apply learned weights and nonlinear activation functions to build progressively more useful internal representations of the input.

The word “hidden” means only that the layer has no target values of its own: the training data specifies inputs and desired outputs, and the network must discover what the intermediate layers should compute. The number and width of hidden layers largely define a model’s capacity, which is why the concept underlies everything from a small multilayer perceptron to a large transformer in artificial intelligence.

How a hidden layer transforms its input

A hidden layer computes a weighted sum plus bias for each of its neurons and then passes the result through an activation function. For a fully connected layer, this is written as h = f(Wx + b), where W is the weight matrix, b is the bias vector and f is a nonlinearity such as ReLU.

The nonlinear activation is essential. Without it, any stack of hidden layers collapses mathematically into a single linear transformation, so depth would add no expressive power. With it, each layer can bend and fold the representation space, and the universal approximation theorem shows that even one sufficiently wide hidden layer can approximate any continuous function on a bounded domain.

The table below summarizes the parameters that define a single hidden layer.

ParameterWhat it controlsTypical range
WidthNumber of neurons or channelsTens to tens of thousands
Activation functionType of nonlinearityReLU, GELU, SiLU, tanh
Weights and biasesLearned mapping from input to outputSet by backpropagation
NormalizationStability of activationsLayer norm, batch norm, RMSNorm
Dropout rateRegularization strength0 to 0.5
Parameters that define a hidden layer

Types of hidden layers

Hidden layers come in several architectural types that differ in how neurons connect to the previous layer. The choice reflects the structure of the data: grids for images, sequences for text and graphs for relational data.

Layer typeConnection patternTypical use
Dense layerEvery input connects to every neuronMultilayer perceptrons, transformer feedforward blocks
Convolutional layerLocal kernel shared across positionsImages, audio, spatial data
Recurrent layerState carried across time stepsSequences in LSTM and GRU models
Attention layerWeights computed from token similarityTransformers and language models
Pooling layerFixed downsampling, no learned weightsReducing spatial resolution
Embedding layerLookup table indexed by token IDMapping discrete symbols to vectors
Common hidden layer types and their typical use

In a transformer, one block typically contains several hidden layers: a self-attention sublayer, a position-wise feedforward sublayer, normalization and residual connections. Counting “layers” in published models therefore usually refers to blocks, not individual sublayers.

What hidden layers learn

Hidden layers learn hierarchical features, with early layers capturing simple patterns and later layers combining them into abstract concepts. In an image classifier, the first convolutional layers respond to edges and color gradients, middle layers respond to textures and parts, and deep layers respond to whole objects.

This representation learning is the main reason deep networks outperform hand-engineered features. The activations of a hidden layer, often called hidden states or embeddings, can be extracted and reused for search, clustering or transfer learning, and interpretability research studies them to find which concepts individual directions encode.

Individual hidden neurons are rarely clean detectors of a single concept. Research on superposition and polysemantic neurons shows that networks often spread one feature across many units and pack several features into one unit, which is why interpretability work increasingly analyzes directions in activation space rather than single neurons.

Choosing depth and width

Choosing hidden layer depth and width is a trade-off between capacity, data and compute. More layers allow more composition of features, while wider layers allow more features per level, but both increase parameter count, memory use and the risk of overfitting on small datasets.

The decision table below gives practical starting points that are usually refined by validation experiments or neural architecture search.

SituationSuggested configurationReason
Small tabular datasetOne or two dense layers, 32 to 256 unitsLimits overfitting
Image classificationDeep convolutional stack with residual connectionsSpatial hierarchy of features
Language modelingMany transformer blocks of equal widthScaling improves quality predictably
Latency-sensitive inferenceFewer, narrower layers or distillationLower compute per prediction
Very deep networkResidual connections and normalizationKeeps gradients stable
Starting points for hidden layer design

As a rule, depth needs supporting techniques. Careful weight initialization, normalization and residual connections are what made networks with dozens or hundreds of hidden layers trainable after the vanishing gradient problem limited earlier deep models.

Common problems with hidden layers

Problems with hidden layers usually appear as training instability or poor generalization. The symptoms are often visible in activation statistics and gradient norms before they show in the final metrics.

SymptomLikely causeTypical fix
Dead neuronsReLU units stuck at zero outputLeaky ReLU, lower learning rate
Vanishing gradientsSaturating activations in deep stacksReLU family, residual connections
Exploding activationsPoor initialization or high learning rateNormalization, gradient clipping
OverfittingToo many hidden units for the dataDropout, weight decay, smaller layers
UnderfittingToo little capacityAdd width or depth
Hidden layer problems, causes and fixes

A frequent misconception is that more hidden layers always help. Beyond the point that the data and training budget can support, extra layers add cost and instability without improving accuracy.

A non-technical view

A neural network is like an assembly line for information. Raw data enters at one end and an answer comes out at the other. The hidden layers are the workstations in between, where the data is reshaped step by step.

Nobody tells these workstations exactly what to do. During training, the network adjusts them on its own until the final answers come out right. One station may learn to notice edges in a photo, the next to spot shapes, and a later one to recognize a face.

A concrete example

A hospital data team builds a model that predicts whether patients will be readmitted within 30 days, using 40 numeric features from discharge records. The first prototype uses a single hidden layer of 512 neurons, and it reaches high training accuracy but performs poorly on held-out patients, a classic sign of overfitting on a dataset of about 20,000 records.

The team switches to two hidden layers of 64 and 32 neurons with ReLU activations, a dropout rate of 0.3 and weight decay. Validation performance improves and the gap between training and test scores narrows. The smaller design also runs fast enough for real-time use at discharge, showing that the right size of hidden layers matters more than raw capacity.