Overfitting

Overfitting is a modeling error that occurs when a machine learning algorithm captures the noise and random fluctuations in the training data to the extent that it negatively impacts the model's ability to generalize to new, unseen data. It happens when a model is too complex relative to the amount and simplicity of the available data, leading it to "memorize" the training set rather than learning the underlying patterns.

This phenomenon is primarily encountered during the training phase of supervised learning. It is the opposite of underfitting, where a model is too simple to capture the trend. The balance between these two states is known as the bias-variance tradeoff: overfitting represents a state of low bias (low error on training data) but high variance (high error on test data).

A concrete example occurs in image recognition when a model is trained on a small set of photos of dogs. If the model overfits, it may learn that a "dog" must always have a green grass background because every training image featured grass. Consequently, the model will fail to identify a dog in a photo taken on a sidewalk, as it has mistaken a coincidental background feature for a defining characteristic of the object.