“We did a re-release of the benchmark, we call it Apex 1.1… number one, made sure the tasks are well specified… number two, we have rubrics that actually penalize this particular behavior.”

"The Cognitive Revolution" · AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?

Play from 1:14:42
1:14:42

Audio streams from the publisher’s own feed. Verbatim does not host or copy it.

What was described

Created Apex 1.1 benchmark by making tasks well-specified and adding rubrics that penalize scatter-gunning answers

Said nearby on the same subject

“We were able to collect a bunch of experts' rubrics and answers, also supplemented those with synthetic data... graded it against the expert answers with real request RL.”
♫
Said on Unsupervised Learning with Jacob Effron
How they work Share this moment
“I have on my website BEP Research, cost per completed task… benchmarks based on the model and the vertical”
♫
Said on Monetary Matters with Jack Farley
How they work Share this moment
“Benchmarking is very powerful, especially to identify friction, slow pages, unnecessary checkouts… you want to benchmark the friction or the basic hygiene stuff, but you want to protect what makes you unique.”
JP
Jean-Christophe Pitié
The Agile Brand with Greg Kihlström® | What CMOs Need to Know About Marketing Technology, AI & CX ·
Why it worked Share this moment
“…pulling in sentry bugs, cataloging them into issues and tasks, completing them and then creating PRs for me.”
AJ
Adam Jacob Swamp Club
Syntax - Tasty Web Development Treats ·
How they work Share this moment

Moments like this, every Monday

The strongest new moments on your topics, in the speakers' own words. Free for three topics.

No card. No password. One-click unsubscribe.

Track a competitor?

Pro watches up to 25 names or phrases and emails you the morning after they are said.

See Pro, $29 a month

Verbatim quotes a short extract and links to the episode; the episode belongs to its publisher. Remove a show.