42 lines
1.4 KiB
Markdown
42 lines
1.4 KiB
Markdown
# MLP Regularization — MNIST
|
|
|
|
## Setup
|
|
|
|
A baseline multilayer perceptron (`Dense(500) → Dense(100) → Dense(10)`) compared against the same architecture with `Dropout(0.2)` added after each hidden layer.
|
|
|
|
## ⚠️ Two variables changed, not one
|
|
|
|
Before looking at the results: the baseline uses SGD at `lr=0.001`, while the dropout run uses SGD at `lr=0.01` — a 10x higher learning rate, in addition to Dropout being added. This means the comparison below shows what happened between two different configurations, not the isolated effect of Dropout. A rigorous ablation would hold the learning rate constant. This is flagged directly rather than presenting a confounded result as clean evidence for Dropout's effect.
|
|
|
|
## Training curves
|
|
|
|
**Baseline:**
|
|
|
|
|
|
|
|
**With Dropout (and higher learning rate):**
|
|
|
|
|
|
|
|
## Where each model gets it wrong
|
|
|
|
**Baseline misclassifications:**
|
|
|
|
|
|
|
|
**Dropout version misclassifications:**
|
|
|
|
|
|
|
|
## Results
|
|
|
|
| Model | Test loss |
|
|
|---|---|
|
|
| Baseline (no dropout, lr=0.001) | 0.279 |
|
|
| Dropout (0.2, lr=0.01) | 0.099 |
|
|
|
|
## Notebooks
|
|
|
|
- [`mnist-mlp-pred.ipynb`](../mnist-mlp-regularization/mnist-mlp-pred.ipynb)
|
|
- [`mnist-mlp-pred-dropout.ipynb`](../mnist-mlp-regularization/mnist-mlp-pred-dropout.ipynb)
|