“You need more pre-training data… give the model a bunch of data and think… transform this data… pre-trained on it would make the model smarter”
♫
Said on "The Cognitive Revolution"
“Maybe if there were fewer of those it would be much better and if this forking really happened at points that were interpretable for people”
Eric Bigelow · "The Cognitive Revolution" · How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning
Play from 1:22:09
1:22:09
Audio streams from the publisher’s own feed. Verbatim does not host or copy it.
What was described
suggest training away sharp forking by reducing biased tokens (e.g., open parenthesis) in the training data
Said nearby on the same subject
“We can train rats at a fifth of the price of a trained demining dog… we have demonstrated… we can reduce the cost price per square meter”
♫
Said on TED Radio Hour
“Train, train, train. Repeat, repeat, repeat… Doing once a year check-the-box training… is ineffective.”
JS
Jake Silverman
Risk Management Show ·
“The tokenizer ... thought that solid gold Magikarp was ... a super common word ... but it's also never seen before in the training data.”
♫
Said on Odd Lots
Moments like this, every Monday
The strongest new moments on your topics, in the speakers' own words. Free for three topics.
No card. No password. One-click unsubscribe.
Track a competitor?
Pro watches up to 25 names or phrases and emails you the morning after they are said.
See Pro, $29 a monthVerbatim quotes a short extract and links to the episode; the episode belongs to its publisher. Remove a show.