// Hacker Noon · 5 March 2026

What Happens if You Remove ReLU From a Deep Neural Network?

Removed ReLU from a 5-layer PyTorch MLP. The model trained without errors, loss decreased every epoch, and it still hit 91.8% on MNIST — matching single-layer logistic regression exactly. Four hidden layers with 575K parameters added zero expressive power. The gradient data was the unexpected part....

Hacker Noon

@hacker-noon · Emmimal P Alexander

hackernoon.com

Hacker Noon@hacker-noon