// Link · 28 February 2026

dReLU Sparsification: Recovering LLM Performance with 150B Token Pretraining

Learn how dReLU-based ReLUfication restores model capabilities for Mistral-7B and Mixtral-47B. Discover the high-quality pretraining datasets and mixture ratios used to achieve elite activation sparsity.

Hacker Noon

@hacker-noon · hackernoon.com

hackernoon.com

Hacker Noon@hacker-noon