“The authors do show that a very small amount of additional post-training on the sparsely activating model is enough to recover a large majority of the performance loss.”
“We saw not just recovery, but even improvement of, you know, 5 to 15 points of performance, which was, like, really surprising.”
Curt Tigges · "The Cognitive Revolution" · The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering
Audio streams from the publisher’s own feed. Verbatim does not host or copy it.
What was described
applied a sparsified, lightly post-trained model and measured performance improvement of 5-15 points on several benchmarks
5 to 15 points
Said nearby on the same subject
“The 30% CGT applies from that point on”
“I applied and for, I have knee issues. I had a knee replacement recently. And I'm telling you within 60 seconds, I was approved for severe pain.”
“We put a ton of time into not just model architecture, but also the data pipelines around it, how we eval the product, and having a really good RL loop around post-training and mid-training the model”
Moments like this, every Monday
The strongest new moments on your topics, in the speakers' own words. Your first brief is free, then USD {{ TIERS.starter.price }} a month for three topics.
No card to start. No password. Cancel any time.
Track a competitor?
Pro watches up to 25 names or phrases and emails you the morning after they are said.
See Pro, $29 a monthVerbatim quotes a short extract and links to the episode; the episode belongs to its publisher. Remove a show.