Technology
A test of AI model performance across the industry is being gamed by technology giants, making objective scientific comparison impossible, researchers have claimed
AI models go head-to-head in Chatbot Arena
Andriy Onufriyenko/Getty Images
An industry-standard league table for ranking artificial intelligence models is being deliberately distorted by technology giants, researchers have claimed, leading to a misleading picture of which AIs are the best.
Sara Hooker at Cohere Labs, a US non-profit, and her colleagues claim to have found that the popular Chatbot Arena benchmark is a “distorted playing field”, with policies that end up giving an advantage to large companies like Meta, Amazon and Google by allowing them to discard models that score poorly.
More from New Scientist
Explore the latest news, articles and features
