1.4 KiB
1.4 KiB
MLP Regularization — MNIST
Setup
A baseline multilayer perceptron (Dense(500) → Dense(100) → Dense(10)) compared against the same architecture with Dropout(0.2) added after each hidden layer.
⚠️ Two variables changed, not one
Before looking at the results: the baseline uses SGD at lr=0.001, while the dropout run uses SGD at lr=0.01 — a 10x higher learning rate, in addition to Dropout being added. This means the comparison below shows what happened between two different configurations, not the isolated effect of Dropout. A rigorous ablation would hold the learning rate constant. This is flagged directly rather than presenting a confounded result as clean evidence for Dropout's effect.
Training curves
Baseline:
With Dropout (and higher learning rate):
Where each model gets it wrong
Baseline misclassifications:
Dropout version misclassifications:
Results
| Model | Test loss |
|---|---|
| Baseline (no dropout, lr=0.001) | 0.279 |
| Dropout (0.2, lr=0.01) | 0.099 |