# Do Brains Have a Learning Rate? An LLM Thought Experiment That Won't Leave My Head
AI-readable version of this post.
Path: /evolutionary-biology/do-brains-have-a-learning-rate
Category: Evolutionary Biology
Date: 2026-08-02
Author: Shubham Blogpost
Reading time: 6 min
Description: Training language models made me wonder whether evolution has been solving the same optimization problem for millions of years.
I spend a lot of time training and fine-tuning language models. Like most people working with LLMs, I've spent hours tweaking learning rates, watching loss curves, restarting runs because the model diverged, and celebrating when a tiny change in hyperparameters suddenly made everything click. Somewhere in the middle of all of that, a strange thought started bothering me. The more I learned about optimization in neural networks, the more I found myself wondering whether evolution had been wrestling with a surprisingly similar problem for millions of years.
One thing that repeatedly shows up in machine learning is that learning too aggressively is often a terrible idea. If you've ever fine-tuned a large language model with an overly high learning rate, you've probably seen it happen. The model adapts incredibly quickly, but it also starts forgetting what it already knew. Instead of carefully integrating new information into a rich internal representation, it bulldozes over months of expensive pre-training. The model hasn't become more intelligent—it has simply become more reactive.
That's why fine-tuning recipes for modern LLMs are usually surprisingly conservative. We don't want the model to reinvent itself every gradient update. We want it to make tiny, precise adjustments while preserving the enormous amount of knowledge it already contains.
```python
# A typical conservative fine-tuning setup for an LLM
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
# Warmup prevents shocking the model early on,
# followed by a slow decay to carefully settle into a good minimum
scheduler = get_cosine_schedule_with_warmup(
optimizer,
num_warmup_steps=100,
num_training_steps=10000
)
```
In a strange way, modern AI isn't rewarded for changing quickly. It's rewarded for changing carefully.
The funny thing is that this isn't just an engineering problem. Biology has faced a remarkably similar challenge for as long as brains have existed. Every organism has to adapt to new experiences without destroying everything it learned before. If every new experience completely rewrote your brain, you'd forget how to walk every time you learned someone's name. Somehow evolution found a way to keep learning while preserving stability, and neuroscience has spent decades trying to understand how that balancing act works.
There's actually a name for this problem: the **stability–plasticity dilemma**.
```mermaid
flowchart TD
A[New Experience] --> B{Learning Rate}
B -- Too High --> C[High Plasticity]
B -- Too Low --> D[High Stability]
B -- Balanced --> E[Optimal Adaptation]
C --> F[Catastrophic Forgetting
Overwrites old memories]
D --> G[Rigidity
Fails to learn new things]
E --> H[Knowledge Integration
Learns without forgetting]
style C fill:#ef4444,color:#fff,stroke:#b91c1c
style D fill:#f59e0b,color:#fff,stroke:#b45309
style E fill:#10b981,color:#fff,stroke:#047857
```
Stability means retaining useful knowledge over long periods of time. Plasticity means remaining flexible enough to learn something new. Too much stability, and you become incapable of adapting. Too much plasticity, and you constantly overwrite yourself. Every intelligent system, whether it's biological or artificial, seems to live somewhere on that spectrum.
That realization made me wonder whether learning rate is just one engineering solution to a much older optimization problem. In machine learning, learning rate determines how aggressively a model updates itself. Human brains obviously don't run gradient descent, but computational neuroscience often models learning with parameters that behave in surprisingly similar ways.
```mermaid
flowchart LR
A((Hippocampus)) -- Fast Learning Rate
High Plasticity --> B(Rapidly records specific episodes
and new events)
C((Neocortex)) -- Slow Learning Rate
High Stability --> D(Gradually extracts general rules
and patterns over time)
B -. Memory Consolidation .-> D
```
Different brain systems appear to update their internal beliefs at different speeds depending on uncertainty, context, and experience. Maybe brains don't have one universal learning rate. Maybe they have many, each serving a different purpose.
This is where my thoughts became much more speculative.
While reading about evolutionary theories of autism, I came across hypotheses like the **Solitary Forager Hypothesis**, which proposes that some autistic traits—such as sustained attention, persistence, deep focus, and strong systemizing—may have been advantageous in certain ancestral environments, particularly those involving hunting or solitary foraging. Whether or not that hypothesis ultimately turns out to be correct isn't really the point. What fascinated me was the broader idea that evolution doesn't necessarily optimize for one universal cognitive style. Different strategies may have been useful under different environmental pressures.
That immediately made me wonder whether the stability–plasticity trade-off might also exist at the level of cognitive styles. Before anyone misunderstands me, this is **not** me saying autism is "a lower learning rate." Human cognition is vastly more complicated than a single optimization parameter, and autism involves differences across many domains of development, perception, communication, attention, and sensory processing. Reducing all of that to one number would be both scientifically inaccurate and unfair.
But thought experiments are often useful precisely because they force us to ask questions we wouldn't normally ask.
Suppose evolution wasn't simply trying to maximize intelligence. Suppose it was trying to maximize adaptability while preserving knowledge. In machine learning, we know that learning too aggressively can destroy previous capabilities. We also know that changing too cautiously can make a model painfully slow to improve. Somewhere between those two extremes lies an optimal balance. It doesn't seem unreasonable to wonder whether evolution has spent millions of years searching for similar balances across different kinds of minds.
Another interesting observation is that the history of AI itself almost looks like a search for better optimization strategies. Early deep learning models often required extremely careful optimization because training was unstable. Modern foundation models still rely on carefully tuned learning-rate schedules, but they also benefit from dramatically better optimizers, normalization techniques, initialization methods, and architectural improvements. We've gradually become better at letting models learn more efficiently without destabilizing them.
That raises another question I keep thinking about. Maybe the future of AI isn't simply about making learning rates smaller or larger. Maybe it's about building optimization algorithms that let models learn quickly **without** forgetting. If that happens, the learning-rate hyperparameter itself may become much less important than the mechanisms surrounding it.
And maybe that's the real lesson.
Perhaps intelligence isn't fundamentally about learning fast.
Perhaps it's about knowing **what deserves to change** and **what deserves to remain stable**.
The more I work with language models, the less I think of intelligence as "absorbing information." Instead, I increasingly think of it as preserving useful structure while making careful updates when reality demands it. That's true for LLMs. It appears to be true for brains. It may even be one of the deepest optimization problems evolution has ever faced.
I don't know whether this analogy ultimately leads anywhere. Maybe it's completely wrong. Maybe neuroscience will eventually show that the comparison breaks down almost immediately. But I enjoy the fact that modern AI keeps generating questions that reach far beyond engineering. Every once in a while, a hyperparameter in a neural network makes you wonder about evolution, cognition, and why different minds exist in the first place.
Those are the kinds of questions that keep me up at night far longer than any training run ever has.
---
## References
Grossberg, S. (1987). *Competitive learning: From interactive activation to adaptive resonance.* This work introduced the stability–plasticity dilemma, a foundational idea in computational neuroscience describing the challenge of learning new information without overwriting existing knowledge.
McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995). *Why There Are Complementary Learning Systems in the Hippocampus and Neocortex.* This influential paper proposes that the brain uses complementary learning systems to balance rapid adaptation with long-term memory.
Kirkpatrick, J., et al. (2017). *Overcoming Catastrophic Forgetting in Neural Networks.* Introduces Elastic Weight Consolidation, one of the landmark methods for continual learning in deep neural networks.
Reser, J. E. (2011). *The Solitary Forager Hypothesis for Autism.* Presents an evolutionary hypothesis suggesting that some autistic traits may have been advantageous in ancestral hunting and foraging environments. This remains a hypothesis rather than scientific consensus.
Goodfellow, I., Bengio, Y., & Courville, A. *Deep Learning.* Chapters on optimization and gradient-based learning provide an excellent introduction to learning rates and neural network training dynamics.
> **Disclaimer:** The comparisons made in this article are speculative thought experiments inspired by machine learning and computational neuroscience. They should not be interpreted as evidence that autism is equivalent to a particular machine-learning hyperparameter or optimization strategy.