“We did a re-release of the benchmark, we call it Apex 1.1… number one, made sure the tasks are well specified… number two, we have rubrics that actually penalize this particular behavior.”
"The Cognitive Revolution" · AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?
Play from 1:14:42
1:14:42
Audio streams from the publisher’s own feed. Verbatim does not host or copy it.
What was described
Created Apex 1.1 benchmark by making tasks well-specified and adding rubrics that penalize scatter-gunning answers
Said nearby on the same subject
“We were able to collect a bunch of experts' rubrics and answers, also supplemented those with synthetic data... graded it against the expert answers with real request RL.”
“Benchmarking is very powerful, especially to identify friction, slow pages, unnecessary checkouts… you want to benchmark the friction or the basic hygiene stuff, but you want to protect what makes you unique.”
JP
Jean-Christophe Pitié
The Agile Brand with Greg Kihlström® | What CMOs Need to Know About Marketing Technology, AI & CX ·