“I used Phoenix, which is a little evals library that you can set up there to run tests, immense real information, and used a large model as a judge to pick the best model that could do the job.”

“Grokbot will also run third-party models to give you the best bang for your buck… SpaceX will use the best back-end model for any given task”

“Our eval harnesses and the library there is absolutely critical to our success”

“We trained for 90 steps, with three seeds for each fine-tuned character model, nine runs in total… using Impossible Bench… broken tests… model could only get the reward if it tampered with the tests.”

“I actually set up some evals and with post-processing, I was able to get Llama 3.2b to perform as well as Cloud Sonnet at summarizing locally the conversations that I was having.”