DeepMind and activation-function 'mining': why it's time to retire ReLU

For decades, we've lived in a world where the ReLU activation function was an unshakable standard. It's as simple as a brick and just effective enough not to get in the way of neural networks learning. But let's be honest: ReLU (Rectified Linear Unit) didn't become popular because it's perfect.

It became popular because in 2012 we didn't have the resources to try anything more complex. Now DeepMind has decided to stop reading tea leaves and has turned the search for mathematical formulas into full-scale industrial mining. The research team built what they call a "computational mine."

The idea is simple and slightly insane at the same time: if we don't know which mathematical function works best for deep learning, let's just try them all. It's classic brute force, taken to the extreme. Instead of making mathematicians spend years deriving elegant proofs, DeepMind threw thousands of GPUs at the problem to "dig up" the ideal algorithm.

Why is this happening now? The LLM market has hit an efficiency ceiling. We keep piling on parameters, but the models' basic building blocks barely change.

DeepMind realized that even a tiny efficiency gain at the activation-function level, at the scale of GPT-4 or Gemini, saves millions of dollars in electricity and weeks of training. This isn't just academic curiosity; it's pure economics. During its "mining," the system tested millions of combinations of mathematical operators.

The researchers looked for functions that not only post high accuracy on paper but also "play nicely" with modern hardware. It turned out that many theoretically strong functions are too expensive to compute on GPUs, which makes them useless in real production. DeepMind aimed for the sweet spot: computational lightness plus mathematical flexibility.

The results are impressive. The discovered functions beat not only good old ReLU but also more modern alternatives like Swish and GeLU. The most interesting part is the shift in approach.

We're moving from an era of "smart people inventing algorithms" to an era of "smart systems growing algorithms." This is the real AutoML people have dreamed about for the past five years, except now it has reached the very foundation of neural connections. What does this mean for the industry?

Most likely, the next generation of large language models will feature architectures that look strange to us. They'll use functions that no sane human would ever derive on a whiteboard, because they don't look "pretty" by the standards of classical calculus. But they will work.

And they'll run faster than anything we've seen before. The bottom line: DeepMind has clearly shown that the AI "gold rush" is moving from giant datasets to reinventing basic math. If you thought the foundations of deep learning were already set in concrete, brace yourself: someone just started taking a jackhammer to them.