Overfitting, Underfitting, and Why Your Model Might Be Lying to You

One of the most frustrating things in machine learning is training a model that looks like it’s working until you try it on new data. Suddenly, predictions that seemed perfect during training fall apart in the real world. Why does this happen? Welcome to the classic machine learning struggle: overfitting and underfitting.

These two concepts are crucial to understanding why your model might seem brilliant one moment and completely clueless the next. They’re tied to something called the bias-variance tradeoff, a fundamental challenge in building models that generalize well.

In this post, we’ll break down what overfitting and underfitting are, how to spot them, and why striking the right balance is key to building reliable machine learning systems.

What is Overfitting?

Imagine you’re studying for an exam. Instead of understanding the subject, you just memorize every answer from past tests. On the day of the exam, any question that looks familiar is easy, but the moment you see something new, you’re lost. That’s overfitting in a nutshell.

In machine learning, overfitting happens when a model learns the training data too well, including its noise, errors, and irrelevant details. It performs brilliantly on the data it’s seen but struggles with new, unseen examples.

An overfitted model is too complex. It might draw very squiggly, unnecessarily complicated decision boundaries just to perfectly classify every point in the training data. While this looks impressive on paper, it usually leads to poor performance on real-world data, where those same quirks don’t exist.

Overfitting is like mistaking coincidence for cause-and-effect. The model becomes so tightly tied to its training set that it loses the ability to generalize.

Signs of Overfitting:

• Very low error on the training data.
• Much higher error on validation or test data.
• Complex models with too many parameters relative to the amount of data.

What is Underfitting?

Now imagine someone preparing for that same exam but barely studying. They skim the material, misunderstand key concepts, and don’t grasp the subject deeply enough. On the test, they struggle with both familiar and unfamiliar questions. That’s underfitting.

Underfitting happens when a model is too simple to capture the underlying patterns in the data. It fails to even perform well on the training set, let alone new data.

This can happen if the model itself isn’t powerful enough (like trying to fit a straight line to data that clearly forms a curve), or if the data hasn’t been properly prepared or cleaned. Underfitting is a sign the model hasn’t been given enough capacity or flexibility to learn from the data.

Signs of Underfitting:

• High error on both training and test data.
• Oversimplified models ignoring important patterns.
• Poor performance from the start, with little improvement during training.

The Bias-Variance Tradeoff

Overfitting and underfitting are two extremes on a spectrum. On one end, you have models that are too flexible (overfitting, high variance), and on the other, models that are too rigid (underfitting, high bias).

• Bias refers to errors introduced by assuming too simple a model. High bias leads to underfitting.
• Variance refers to how sensitive the model is to the specific data it was trained on. High variance leads to overfitting.

The key is finding the sweet spot where your model balances low bias and low variance, complex enough to capture real patterns but not so flexible that it memorizes noise.

Why This Matters in Practice

Understanding overfitting and underfitting isn’t just academic. It’s the foundation of building models that actually work outside of ideal lab conditions.

Think about a model for medical diagnoses. If it’s overfitted to one specific dataset, it might make mistakes on patients from a different hospital. If it’s underfitted, it might miss obvious signs of illness even in the training data. Neither scenario is acceptable.

This is why machine learning isn’t just about getting the best performance on your training set. It’s about building models that generalize well to the real world.

How to Handle Overfitting and Underfitting

Though we’ll cover techniques in more detail in future posts, here’s a high-level view of strategies to tackle these issues:

To reduce overfitting:

• Use simpler models.
• Collect more data.
• Apply regularization techniques (like L1/L2 penalties).
• Use dropout (for neural networks).
• Evaluate on validation sets and stop training early if performance worsens.

To reduce underfitting:

• Use more complex models.
• Improve feature engineering.
• Train longer or adjust learning rates.
• Reduce regularization if it’s too aggressive.

Machine learning isn’t just about algorithms, it’s about knowing when to adjust, simplify, or complicate your approach based on what the data tells you.

What’s Next?

In this post, we unpacked the reasons why machine learning models sometimes deceive us into thinking they’re better than they really are. Overfitting and underfitting are two of the most common, and most fixable, challenges in ML.

Next time, we’ll continue deepening our understanding by looking at how to prepare data effectively, and why your model’s performance often depends less on the algorithm you choose and more on the data you feed it.

Stay tuned!