
XYBench: Why AI Assistants Miss the Real Question You Ask
Ai2's XYBench tests 8,115 misguided questions to see if AI assistants spot the real problem. The award-winning benchmark shows where models can improve.
Anyone who has worked a help desk knows the XY problem: someone wants to do X, assumes Y is the way, and asks only about Y. The best helpers notice and redirect. On November 13, the Allen Institute for AI (Ai2) released XYBench, a benchmark that measures whether AI assistants can do the same, and the results point to a clear, achievable next step for the whole field.
- Size: 8,115 real-world questions from Stack Overflow, Stack Exchange, WikiHow-derived prompts and hand-curated examples
- Recognition: XYBench won the Best Resource Paper Award at EMNLP 2026, according to Ai2
- Partners: built with researchers at Carnegie Mellon, NVIDIA and Johns Hopkins
- Models tested: Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro
What Is the XY Problem in AI Assistants?
Ai2's own example is delightful: someone asks how to thicken frosting that keeps sliding off cupcakes, when the real issue is that the cupcakes are still too warm. A truly helpful AI assistant should suggest letting them cool, not hand over a stiffer frosting recipe. XYBench labels each question with the stated request (Y) and the intended goal (X), filtering out cases where simply answering Y would solve the problem.
The benchmark scores three behaviors separately. Pragmatic solution checks whether a response addresses Y, X, or both. Pragmatic emphasis asks which one the answer actually focuses on. Misconception identification checks whether the model gently steers the user away from the mistaken approach. Claude Sonnet 4.5 serves as the automated judge, and Ai2 validated it against two human reviewers, with agreement between 74.2% and 100% depending on the metric.
How Did the Models Score on XYBench?
According to Ai2's results, every model was more likely to solve the stated request than the underlying goal. On Stack Overflow questions, Y-solutions appeared in 75 to 86% of responses versus 59 to 71% for X-solutions, and models made the real goal the main focus in only 35 to 41% of answers. All five trailed human reference answers on spotting misconceptions.
The most encouraging finding is the gap between judgment and generation. When the same models were shown two candidate answers, one literal and one goal-focused, all six tested (adding GPT-4.1) picked the goal-focused answer 71 to 84% of the time. In other words, the models already recognize better advice; they just do not produce it by default yet. An oracle experiment that handed models the user's true goal helped include more X-solutions, though it did not fully close the gap, which tells researchers that detecting intent is only half the job.
Why This AI Benchmark Matters
Most evaluations reward getting the asked question right. XYBench rewards being genuinely useful, which is a harder and more human target. Because the dataset, paper and code are public, model builders can now measure whether prompting or training changes make assistants better at asking, in effect, what are you really trying to do? It joins a strong run of practical Ai2 evaluations, including StateBench for long-document state tracking.
Expect XYBench scores to start showing up in model cards. When they do, the improvements will be felt by everyone who has ever asked the wrong question for the right reason. For more research breakthroughs, see our AI news section.
Sources: Ai2 on Hugging Face: XYBench — November 13, 2026.
More AI Stories

Gemini Google Wallet Integration: AI Budgeting Explained
Gemini can now read your Google Wallet spending to chart budgets, flag pricier subscriptions and plan holiday costs. Here's how the US rollout works.

SOV-Transcribe-1: Salesforce's Open Speech Model Explained
Salesforce's SOV-Transcribe-1 is an Apache 2.0 open-weight speech-to-text model with a 2.28% word error rate, style steering and 7 languages.

Google AI Edge Gallery Adds Offline Speech: What Devs Get
Google AI Edge Gallery now runs speech recognition and text-to-speech fully on device, with 100+ languages and replies that start in about 100 ms.
