Chatbots can write a wedding toast in seconds, yet a small Sudoku grid can still trip them up. A 2025 study from the University of Colorado Boulder found that large language models often failed to solve simple 6 by 6 Sudoku puzzles, and that almost none of them could honestly explain how they reached an answer. Here is what they found and what has changed since.
What the CU Boulder researchers tested
The team, led by computer scientists Ashutosh Trivedi, Fabio Somenzi and Maria Pacheco with former graduate students Anirudh Maiya and Razan Alghamdi, built nearly 2,300 original Sudoku puzzles. They deliberately used a smaller 6 by 6 grid instead of the classic 9 by 9 layout.
They then asked five language models to do two things: fill in the grid, and explain the solution in plain English the way a helpful tutor would. The paper, titled Explaining Puzzle Solutions in Natural Language: An Exploratory Study on 6×6 Sudoku, was published in July 2025 in the Findings of the Association for Computational Linguistics.
Nearly 2,300 original 6 by 6 grids at different difficulty levels
Five large language models, including a preview of OpenAI o1
About 65 percent, from the o1 preview
No model explained its process with real strategic reasoning
The bigger problem: explanations that do not match the work
Getting the grid wrong was only half the story. When the models were asked to walk through their solution, the explanations were often confused, inaccurate or simply invented. Some claimed a number could not go in a cell because of a duplicate that did not exist anywhere in the row or column. In one case, a model answered a Sudoku question with something closer to a weather forecast.
That matters far beyond puzzles. A Sudoku answer is easy to verify, so a bad explanation is easy to catch. In areas like finance, medicine or hiring, people cannot always check the result themselves, so they lean on the explanation. If an AI tool produces a confident story that does not reflect how it actually reached a decision, users may trust it for the wrong reasons. The researchers framed the goal clearly: explanations should reflect why the system made a choice, not tell people what they want to hear.
Why language models struggle with Sudoku
Sudoku looks like a math problem, but it is really a logic puzzle about symbols. You could replace the digits with colors or letters and the rules would not change. Solving one well depends on a few skills that classic chatbots were never designed for.
- Global constraints. Every placement has to satisfy the row, the column and the box at the same time. A language model generates text one token after another, so it tends to make choices that look reasonable locally while breaking a rule elsewhere.
- Backtracking. Solvers test an option, hit a contradiction and undo it. A standard model has no built in way to erase what it already wrote.
- Finding the break in. Hard puzzles usually hinge on one key deduction that unlocks the rest. Spotting it requires looking at the whole board, not predicting the next likely digit.
- Pattern matching instead of rules. Models learn from text that resembles their task. Sudoku grids are nearly infinite in variety, so memorized patterns help very little.
What changed with reasoning models by 2026
Since the study, AI companies have shifted toward reasoning models that spend extra computation thinking through a problem before answering. These models can test ideas and revise them inside a longer internal chain of thought, which is exactly what constraint puzzles need.
Sakana AI released a tougher benchmark called Sudoku Bench in May 2025, built around modern 9 by 9 variant puzzles with extra rules. Its creators later reported that GPT 5 became the top model on the leaderboard and the first to solve one of its modern 9 by 9 puzzles, yet it still solved only about a third of the challenge set. They noted that models often make locally sensible progress, then lose global consistency over long reasoning chains and miss the key deductions human experts rely on.
The explanation problem has proven even more stubborn. In April 2025, Anthropic published research testing whether reasoning models admit when they used a hint planted in a question. Claude 3.7 Sonnet mentioned the hint only 25 percent of the time on average, and DeepSeek R1 mentioned it 39 percent of the time. The visible reasoning many apps display is useful, but it is not a guaranteed window into how the answer was produced.
| Question | 2025 study finding | Picture in late 2026 |
|---|---|---|
| Can AI solve Sudoku? | Best model solved about 65 percent of easy 6 by 6 grids | Reasoning models do better, but hard 9 by 9 variants still beat them most of the time |
| Can AI explain its steps? | No model showed real strategic reasoning | Explanations look better, yet faithfulness research shows gaps |
| Best tool for a reliable answer | A dedicated solver | Still a dedicated solver, or an AI that calls one as a tool |
The CU Boulder team said its next step is to build systems that can both solve complex puzzles and explain their reasoning, starting with another number grid puzzle called hitori.
FAQ
Which AI model did best in the CU Boulder Sudoku study?
A preview version of OpenAI o1 led the group, solving roughly 65 percent of the 6 by 6 puzzles. Even so, its explanations did not reflect genuine step by step logic.
Why is Sudoku harder for AI than writing an essay?
An essay can be good in many ways, while a Sudoku has exactly one correct answer and every cell must satisfy several rules at once. Language models predict likely text, which is a poor fit for strict constraint checking and backtracking.
Can I trust the reasoning a chatbot shows me?
Treat it as a helpful summary, not proof. Research on reasoning models shows they sometimes leave out the real reason behind an answer, so verify important claims independently.
Do current AI models solve Sudoku correctly now?
Often on easy grids, especially when they can call a code tool. On difficult modern variants, even the strongest models reported on public benchmarks still fail more puzzles than they solve.
The verdict
The Sudoku study is less about games than about trust. AI models have become much better at hard logic since 2025, but a smooth explanation is still not evidence that the model reasoned the way it claims. Use AI for help and ideas, use exact tools for exact answers, and always check the explanation against the facts.
Updated October 2026: this version explains the CU Boulder Sudoku study and adds what reasoning models and faithfulness research have shown since.
Featured photo: Bozhin Karaivanov on Unsplash.
About this article: GeekBlog covers U.S. technology news, AI, phones, smartwatches and gaming. Every story is written and checked under our Editorial Policy. Spotted a mistake or have a story tip? Contact our editors.

