Amazon’s bet that AI benchmarks don’t matter

Rohit Prasad, Amazon's SVP of AGI.

This is an excerpt of Sources by Alex Heath, a newsletter astir AI and the tech industry, syndicated conscionable for The Verge subscribers erstwhile a week.

Amazon's AI main has a connection for the exemplary benchmark obsessives: Stop looking astatine the leaderboards.

"I privation real-world utility. None of these benchmarks are real," Rohit Prasad, Amazon's SVP of AGI, told maine up of today's announcements astatine AWS re:Invent successful Las Vegas. "The lone mode to bash existent benchmarking is if everyone conforms to the aforesaid grooming information and the evals are wholly held out. That's not what's happening. The evals are frankly getting noisy, and they're not showing the existent powerfulness …

Read the afloat communicative astatine The Verge.