Regularization

Regularization adds a preference for simpler solutions during training. Without it, a model with enough parameters can memorize the training dataset — achieving zero training loss while performing poorly on new data. Regularization penalizes complexity, nudging the optimizer toward solutions that generalize.

## The two main forms

**L1 regularization** (lasso) adds the sum of the absolute values of the model weights to the loss function. This drives many weights to exactly zero, performing implicit feature selection. Models trained with L1 tend to be sparse — most features are ignored. L1 is useful when you suspect only a few input features actually matter.

**L2 regularization** (ridge / weight decay) adds the sum of the squared values of the weights. This drives weights toward zero but rarely to exactly zero — all features remain in the model, contributing small amounts. L2 is the default regularizer in most deep learning frameworks (`weight_decay` in PyTorch, `kernel_regularizer=l2(λ)` in Keras).

**Elastic net** combines both L1 and L2, balancing feature selection and stability.

## Regularization beyond L1/L2

Modern deep learning uses additional regularization techniques:

- **Dropout**: randomly zeroing a fraction of activations during training. At inference, weights are scaled down by the dropout rate. Forces the network to develop redundant representations.
- **Early stopping**: monitor validation loss and halt training when it begins to rise. The cheapest regularizer — no compute overhead, no hyperparameters to tune.
- **Data augmentation**: artificially expanding the training set (image flips, crops, noise injection) improves generalization without changing the loss function.
- **Batch normalization**: normalizes activations within each batch, which has an implicit regularization effect (though it is primarily used for training stability).

## Choosing the right regularization

The optimal regularization strength (λ) is a hyperparameter — tuned via cross-validation or a validation sweep. Too little: the model overfits. Too much: the model underfits (ignores informative features). In practice:

- Start with L2 (weight decay) as a default.
- Add dropout for large fully-connected layers.
- Use early stopping in all training runs.
- Reserve L1 or elastic net for high-dimensional sparse problems (text classification, genomics).

For enterprise AI teams building production models on tabular business data, aggressive regularization is often more important than architecture choices. Regularized linear models frequently outperform unregularized neural networks on small structured datasets.