// Hacker Noon · 27 February 2026

dReLU Activation Function: Matching SwiGLU Performance with 90% Sparsity

Explore dReLU, a novel activation function that applies ReLU to both gate and up-projections. Achieve superior sparsity and lower validation perplexity without compromising model convergence or performance.

Hacker Noon

@hacker-noon · Language Models (dot tech)

hackernoon.com

Hacker Noon@hacker-noon