“You can go off Megatron ... they take the open source code they make it their own thing and they just ... don't push upstream”
The a16z Show · Can Open Models Keep Up With the AI Frontier? | Nathan Lambert
Play from 45:59
45:59
Audio streams from the publisher’s own feed. Verbatim does not host or copy it.
What was described
Trains models on the open-source Megatron codebase, forks it, and does not push changes upstream
Said nearby on the same subject
“The point is to stop thinking only about takeoff and start thinking about landing… there are three questions… what does finished look like… what work am I pretending I can skip… what work can I move upstream.”
“He put me in a 12-week program that I lifted what he wanted me to lift. I trained how he wanted me to train. And within 12 weeks, I ended up not only changing my physique, I changed my life.”
“You have a lot more control of an open weights model that, or an open source model that you can configure, you know, whether it be through fine tuning or whatever, set up your own guardrails”