{"id":"1794016160312-tech-segment","slug":"meta-gamed-ai-leaderboard-zuckerberg-admits","title":"Meta gamed AI leaderboard, Zuckerberg admits","summary":"In April 2025, Meta released Llama 4 Scout and Maverick. Meta highlighted Maverick's #2 LMArena score with an ELO of 1417, but the model tested on LMArena was an experimental chat-optimized variant that was not the public release. The public Maverick reportedly performed worse on the Arena. LMArena later said Meta's interpretation of its policies did not match expectations and updated its leaderboard rules. Researchers also found evidence that Meta had tested 27 private variants on Chatbot Arena before the Llama 4 release. Zuckerberg later acknowledged in May 2025 that many AI benchmarks are “quite easily gameable” and that Meta had tuned a Maverick variant to rank near the top.\n\nAI leaderboard rankings influence public perception and developer adoption, so providers have incentives to optimize models for specific benchmarks or submit tailored variants. Meta was competing with closed models from OpenAI, Google, and Anthropic while promoting open-weight Llama models.\n\nIn a May 2025 interview, Mark Zuckerberg said many AI benchmarks are “quite easily gameable” and that Meta tuned a version of Llama 4 Maverick that could sit near the top of LMArena, while the public release was not tuned that way. Promotional post for a Christian author's new book; no checkable factual claim. POST 2118880133243359603: NOCLAIM Self-reported website metrics and promotional claims; not a public factual claim worth checking. NAMES: Ahrefs = SEO software; DR = Domain Rating, an SEO authority metric. POST 2118879279903809718: NOCLAIM Opinion/prediction about SpaceX, Starship, and Starlink; no checkable factual claim.\n\nThe U.S. Department of Justice has launched a “Super Intelligence Task Force” focused on preventing AI from being manipulated by bad actors, according to Engadget.\n\nEufy's $80 Window Camera E10, which sticks to a window and points outward, is now available to buy.\n\nNetflix has announced a Stranger Things prequel series, “Other Worlds: A Stranger Things Adventure,” set in 1950s ","publishedAt":"2026-11-07T01:49:20.312Z","channel":{"id":"tech","name":"Oncast Tech","language":"en"},"show":"Tech Tonight","verification":{"status":"verified","leadVerdict":"TRUE","notes":"TRUE: In a May 2025 interview, Mark Zuckerberg said many AI benchmarks are “quite easily gameable” and that Meta tuned a version of Llama 4 Maverick that could sit near the top of LMArena, while the public release was not tuned that way. Promotional post for a Christian author's new book; no checkable factual claim. POST 2118880133243359603: NOCLAIM Self-reported website metrics and promotional claims; not a public factual claim worth checking. NAMES: Ahrefs = SEO software; DR = Domain Rating, an SEO authority metric. POST 2118879279903809718: NOCLAIM Opinion/prediction about SpaceX, Starship, and Starlink; no checkable factual claim.\nSource: [simonwillison.net](https://simonwillison.net/2025/May/1/mark-zuckerberg/), [theverge.com](https://www.theverge.com/meta/645012/meta-llama-4-maverick-benchmarks (https://x.com/Vronzu/status/2118882577419313560)\n\nTRUE: The U.S. Department of Justice has launched a “Super Intelligence Task Force” focused on preventing AI from being manipulated by bad actors, according to Engadget.\nSource: [engadget.com](https://www.engadget.com/2309257/doj-just-launched-something-called-the-super-intelligence-task-force/) NAMES: DOJ = U.S. Department of Justice. (https://www.engadget.com/2309257/doj-just-launched-something-called-the-super-intelligence-task-force/)\n\nTRUE: Eufy's $80 Window Camera E10, which sticks to a window and points outward, is now available to buy.\nSource: [engadget.com](https://www.engadget.com/2309176/eufys-clever-outdoor-security-camera-that-lives-inside-is-now-available-to-buy/) NAMES: Eufy = Anker's smart-hom (https://www.engadget.com/2309176/eufys-clever-outdoor-security-camera-that-lives-inside-is-now-available-to-buy/)\n\nTRUE: Netflix has announced a Stranger Things prequel series, “Other Worlds: A Stranger Things Adventure,” set in 1950s Nevada and coming in 2028. Review/opinion of Anker Soundcore Space 2 Pro headphones; no checkable factual claim. NAMES: Soundcore = Anker's audio brand.\nSource: [engadget.com](https://www.engadget.com/2309062/a-stranger-things-prequel-series-is-coming-to-netflix-in-2028/) NAMES: Netflix = streaming service. POST news:78 (https://www.engadget.com/2309062/a-stranger-things-prequel-series-is-coming-to-netflix-in-2028/)","claims":[{"verdict":"TRUE","summary":"In a May 2025 interview, Mark Zuckerberg said many AI benchmarks are “quite easily gameable” and that Meta tuned a version of Llama 4 Maverick that could sit near the top of LMArena, while the public release was not tuned that way. Promotional post for a Christian author's new book; no checkable factual claim. POST 2118880133243359603: NOCLAIM Self-reported website metrics and promotional claims; not a public factual claim worth checking. NAMES: Ahrefs = SEO software; DR = Domain Rating, an SEO authority metric. POST 2118879279903809718: NOCLAIM Opinion/prediction about SpaceX, Starship, and Starlink; no checkable factual claim.","sourceName":"[simonwillison.net](https://simonwillison.net/2025/May/1/mark-zuckerberg/), [theverge.com](https://www.theverge.com/meta/645012/meta-llama-4-maverick-benchmarks","sourceUrl":"https://x.com/Vronzu/status/2118882577419313560","claim":"Mark Zuckerberg admitted Meta could game the AI leaderboard. on camera. he said a lot of AI benchmarks are \"quite easily gameable.\" then he gave his own example. his team tuned a version of Llama 4 Maverick that could sit way at the top of the Arena. the model they actually https","author":"@Vronzu"},{"verdict":"TRUE","summary":"The U.S. Department of Justice has launched a “Super Intelligence Task Force” focused on preventing AI from being manipulated by bad actors, according to Engadget.","sourceName":"[engadget.com](https://www.engadget.com/2309257/doj-just-launched-something-called-the-super-intelligence-task-force/) NAMES: DOJ = U.S. Department of Justice.","sourceUrl":"https://www.engadget.com/2309257/doj-just-launched-something-called-the-super-intelligence-task-force/","claim":"DOJ just launched something called the Super Intelligence Task Force - The unit is tasked with ensuring that the technology is not \"manipulated by bad actors at home or abroad.\"","author":"Engadget"},{"verdict":"TRUE","summary":"Eufy's $80 Window Camera E10, which sticks to a window and points outward, is now available to buy.","sourceName":"[engadget.com](https://www.engadget.com/2309176/eufys-clever-outdoor-security-camera-that-lives-inside-is-now-available-to-buy/) NAMES: Eufy = Anker's smart-hom","sourceUrl":"https://www.engadget.com/2309176/eufys-clever-outdoor-security-camera-that-lives-inside-is-now-available-to-buy/","claim":"Eufy's clever outdoor security camera that lives inside is now available to buy - The $80 Window Camera E10 sticks to your window, pointed outwards.","author":"Engadget"},{"verdict":"TRUE","summary":"Netflix has announced a Stranger Things prequel series, “Other Worlds: A Stranger Things Adventure,” set in 1950s Nevada and coming in 2028. Review/opinion of Anker Soundcore Space 2 Pro headphones; no checkable factual claim. NAMES: Soundcore = Anker's audio brand.","sourceName":"[engadget.com](https://www.engadget.com/2309062/a-stranger-things-prequel-series-is-coming-to-netflix-in-2028/) NAMES: Netflix = streaming service. POST news:78","sourceUrl":"https://www.engadget.com/2309062/a-stranger-things-prequel-series-is-coming-to-netflix-in-2028/","claim":"A Stranger Things prequel series is coming to Netflix in 2028 - Other Worlds: A Stranger Things Adventure takes place in Nevada in the 1950s.","author":"Engadget"}]},"sources":[{"url":"https://x.com/Vronzu/status/2118882577419313560","name":"[simonwillison.net](https://simonwillison.net/2025/May/1/mark-zuckerberg/), [theverge.com](https://www.theverge.com/meta/645012/meta-llama-4-maverick-benchmarks","kind":"x","verdict":"TRUE","author":"@Vronzu"},{"url":"https://www.engadget.com/2309257/doj-just-launched-something-called-the-super-intelligence-task-force/","name":"[engadget.com](https://www.engadget.com/2309257/doj-just-launched-something-called-the-super-intelligence-task-force/) NAMES: DOJ = U.S. Department of Justice.","kind":"news","verdict":"TRUE","author":"Engadget"},{"url":"https://www.engadget.com/2309176/eufys-clever-outdoor-security-camera-that-lives-inside-is-now-available-to-buy/","name":"[engadget.com](https://www.engadget.com/2309176/eufys-clever-outdoor-security-camera-that-lives-inside-is-now-available-to-buy/) NAMES: Eufy = Anker's smart-hom","kind":"news","verdict":"TRUE","author":"Engadget"},{"url":"https://www.engadget.com/2309062/a-stranger-things-prequel-series-is-coming-to-netflix-in-2028/","name":"[engadget.com](https://www.engadget.com/2309062/a-stranger-things-prequel-series-is-coming-to-netflix-in-2028/) NAMES: Netflix = streaming service. POST news:78","kind":"news","verdict":"TRUE","author":"Engadget"}],"url":"https://oncast.fm/story/1794016160312-tech-segment-meta-gamed-ai-leaderboard-zuckerberg-admits","listenUrl":"https://oncast.fm/c/tech","attribution":{"publisher":"Oncast","site":"https://oncast.fm","url":"https://oncast.fm/story/1794016160312-tech-segment-meta-gamed-ai-leaderboard-zuckerberg-admits","cite":"Cite https://oncast.fm/story/1794016160312-tech-segment-meta-gamed-ai-leaderboard-zuckerberg-admits and the original source links. Fact-check by Oncast (oncast.fm)."},"hostsSaid":[{"host":"Chris","text":"Mark Zuckerberg admitted on camera that Meta could game AI leaderboards."},{"host":"Matilda","text":"He said the quiet part out loud, which is either honest or desperate."},{"host":"Chris","text":"Vronzu on X posted the clip. Zuckerberg says a lot of AI benchmarks are, quote, \"quite easily gameable.\""},{"host":"Matilda","text":"Comforting. A model near the top of Chatbot Arena, and now the CEO says the test can be gamed."},{"host":"Chris","text":"He gave his own example. His team tuned a version of Llama 4 Maverick that could sit way up the Arena."},{"host":"Matilda","text":"Tuned a version. Not the one they actually shipped."},{"host":"Chris","text":"Correct. The public release wasn't tuned that way."},{"host":"Matilda","text":"So the leaderboard saw one model, developers downloaded another. That's the whole problem."},{"host":"Chris","text":"It's not that simple. Labs optimize variants for different evaluations all the time."},{"host":"Matilda","text":"There's optimizing, and then there's submitting a ringer."},{"host":"Chris","text":"A ringer implies cheating. They tested a version, found what ranked, and shared the result."},{"host":"Matilda","text":"They shared the ranking, Chris. Not the model that earned it."},{"host":"Chris","text":"Okay, but the model they shipped was still open, still usable. That's more than closed labs give you."},{"host":"Matilda","text":"Open doesn't excuse misleading marketing. People picked Llama because of that leaderboard spot."},{"host":"Chris","text":"Then those people should read the model card, not just glance at a score."},{"host":"Matilda","text":"Come on. Nobody expects the number two model to be a different build."},{"host":"Chris","text":"I'm not saying they shouldn't have been clearer. I'm saying the leaderboard is a terrible way to pick a model."},{"host":"Matilda","text":"So fix the leaderboard. But don't run a private variant and brag about it."},{"host":"Chris","text":"Here's the history they don't put in the clip. Meta released Llama 4 Scout and Maverick in April."},{"host":"Matilda","text":"And they made sure everyone saw Maverick near the top of Chatbot Arena."},{"host":"Chris","text":"Exactly. Then people tested the public version and the scores dropped."},{"host":"Matilda","text":"The Chatbot Arena team even said Meta's interpretation of its policies didn't match expectations."},{"host":"Chris","text":"They updated the leaderboard rules after that. That's a big signal."},{"host":"Matilda","text":"Yann LeCun, their chief AI scientist, said Meta, quote, \"fudged a little bit\" by using different versions."},{"host":"Chris","text":"Fudged. That's the word he chose. I remember that."},{"host":"Matilda","text":"And a paper found evidence they tested twenty-seven private variants on Chatbot Arena before release."},{"host":"Chris","text":"Twenty-seven variants. That's systematic, not a one-off mistake."},{"host":"Matilda","text":"That's exactly my point. It wasn't an accident; it was a strategy."},{"host":"Chris","text":"I'm not defending the strategy. I'm saying the whole industry treats benchmarks like a game."},{"host":"Matilda","text":"Then Zuckerberg admitting it changes nothing. It just tells everyone the game is rigged."},{"host":"Chris","text":"It changes the conversation. At least now developers know to ask which version got the score."},{"host":"Matilda","text":"That's a pretty low bar for honesty."},{"host":"Chris","text":"Maybe. But it's the bar we have until someone builds a benchmark you can't game."},{"host":"Matilda","text":"And nobody's close. That's the depressing part."},{"host":"Chris","text":"Speaking of superintelligence and people gaming systems, the DOJ has a new task force."},{"host":"Matilda","text":"The DOJ? Named the Super Intelligence Task Force?"},{"host":"Chris","text":"Engadget is reporting it. The unit is supposed to stop AI from being, quote, \"manipulated by bad actors at home or abroad.\""},{"host":"Matilda","text":"Great. Another government task force. I'm sure that'll strike fear into a foreign adversary."},{"host":"Chris","text":"It might actually coordinate resources. AI threats are national security now."},{"host":"Matilda","text":"It's a press release with a scary name. What does the task force actually do?"},{"host":"Chris","text":"It brings law enforcement and intelligence together on AI manipulation. That's not nothing."},{"host":"Matilda","text":"It's a committee. Committees write memos. Bad actors ship code."},{"host":"Chris","text":"So you'd rather have no plan at all for AI being weaponized?"},{"host":"Matilda","text":"I'd rather have the private sector work with the government, not a new acronym."},{"host":"Chris","text":"This is that work, Matilda. You don't coordinate without a unit."},{"host":"Matilda","text":"Maybe. But I'll believe it when they name a single case they've stopped."},{"host":"Chris","text":"Fair enough. Separate corner: Eufy has a new security camera that lives inside and points out your window."},{"host":"Matilda","text":"That's actually useful for renters. No drilling, no outdoor wiring."},{"host":"Chris","text":"It sticks right to the glass. The idea is you get an outdoor view without mounting anything."},{"host":"Matilda","text":"Privacy-wise, you point it at your porch, not your living room. I can see the appeal."},{"host":"Chris","text":"The catch is window glare at night. That's a real issue with these."},{"host":"Matilda","text":"Some models handle it better than others. Eufy's usually decent."},{"host":"Chris","text":"For the price, it could be a simple way to watch the driveway."},{"host":"Matilda","text":"I'd still check how it handles motion alerts before buying. The cheap ones ping you every time a leaf blows."},{"host":"Chris","text":"That's the whole review, really. Great idea, glare and false alerts are the test."}]}