41 lines
1.4 KiB
Markdown
41 lines
1.4 KiB
Markdown
# MLP Regularization — MNIST
|
|
|
|
Comparing a baseline multilayer perceptron against a Dropout-regularized version on MNIST digit classification.
|
|
|
|
## Architecture
|
|
|
|
Both models share the same base architecture:
|
|
|
|
```
|
|
Dense(500, relu) → Dense(100, relu) → Dense(10, softmax)
|
|
```
|
|
|
|
The dropout version adds `Dropout(0.2)` after each hidden layer.
|
|
|
|
## ⚠️ Honest note on this comparison
|
|
|
|
This is **not a clean ablation** — two things changed between the baseline and dropout runs, not one:
|
|
|
|
| | Baseline | Dropout version |
|
|
|---|---|---|
|
|
| Dropout | None | 0.2 after each hidden layer |
|
|
| Optimizer | SGD, lr=0.001 | SGD, lr=0.01 |
|
|
|
|
The learning rate is 10x higher in the dropout run. That means the test loss improvement below **can't be attributed to Dropout alone** — the higher learning rate is a confound. Both notebooks are included as-is because they reflect real experimentation, but the results should be read as "these two configurations produced these two outcomes," not as proof that Dropout alone caused the improvement. A clean ablation would hold the learning rate fixed and vary only Dropout.
|
|
|
|
## Results
|
|
|
|
| Model | Test loss (categorical crossentropy) |
|
|
|---|---|
|
|
| Baseline (no dropout, lr=0.001) | 0.279 |
|
|
| Dropout (0.2, lr=0.01) | 0.099 |
|
|
|
|
## Tech stack
|
|
|
|
TensorFlow / Keras, NumPy, matplotlib
|
|
|
|
## Notebooks
|
|
|
|
- [`mnist-mlp-pred.ipynb`](mnist-mlp-pred.ipynb) — baseline
|
|
- [`mnist-mlp-pred-dropout.ipynb`](mnist-mlp-pred-dropout.ipynb) — with Dropout
|