AI Reasoning and Chain of Thought: Inside the Large Language Model Debate with John Pavlis

Overview

In this episode, Sameer Patel speaks with John Pavlis about AI reasoning, the idea of chain of thought in large language models, and how researchers are testing what these models are really doing inside. The conversation covers definitions of reasoning, how early LLMs differed from larger reasoning models, and what midstream findings suggest about the reliability and nature of their reasoning traces.

Key insights

• Definition and function: AI reasoning is discussed as more than predicting the next word, focusing on intermediate steps or prompts that guide the model to a complex answer.
• Two big debates: whether chain of thought traces are faithful representations of the internal process, or merely decorative tokens with little causal role.
• Empirical findings: studies show brittleness and failures, including many cases where reasoning traces do not causally determine the output.
• Practical implications: the meaning of reasoning affects trust, interpretability, and how researchers treat AI tools as scientific instruments.

Introduction and context

The podcast opens with a broad question: as AI models, particularly large language models, become deeply integrated into everyday tasks and high-stakes scientific work, what exactly are they doing when they appear to reason? John Pavlis, a regular Quanta contributor with a decade of AI reporting, sits down with Sameer Patel to unpack the evolving debate about AI reasoning. They discuss how researchers, journalists, and practitioners have framed the question differently as AI systems transition from simple next-word predictors to tools that can tackle multi-step problems. The episode places Pavlis’s Qualia essay within a broader conversation about whether these models are truly “reasoning” in any human sense or whether they merely imitate the appearance of reasoning through statistical patterns and self-prompting.

Defining reasoning in AI

The hosts take care to distinguish two related ideas: human reasoning, which arrives at conclusions through connected and justifiable steps, and the AI variant, where the model generates a stream of self-generated prompts that it uses to arrive at or approximate an answer. Pavlis explains that AI reasoning is a term of art used to describe the model’s ability to chain intermediate steps that logically lead to an answer, but he emphasizes this is a moving target. The most common label in the industry for this process is chain of thought, sometimes referred to as chain-of-thought reasoning or intermediate tokens, though researchers also use the more neutral term intermediate tokens. The distinction between LLMs and LRMs is clarified: LRMs are trained to produce a sequence of tokens, including internal prompts that guide the solution process, whereas the older LLMs did not explicitly optimize for such intermediate reasoning steps.

Historical emergence and shifting attitudes

The episode recounts the emergence of reasoning capabilities in AI around 2024 with models like 01 from OpenAI and the initial debate about whether reasoning claims were marketing terms or genuine capabilities. Pavlis recalls Melanie Mitchell’s early critiques and notes that the initial reaction in the community was to use air quotes around AI reasoning. Over time, “air quotes” were largely dropped as models demonstrated tangible improvements in solving multi-step problems and even producing frontier-level research results in mathematics, which a four-year cycle ago would have seemed unlikely. The shift in attitude is framed as part of the broader maturation of AI tools from novelty to foundational research instruments.

Ground truth about what the models do

In explaining how these models operate, Pavlis and Patel describe the chain-of-thought behavior as a series of self-generated prompts the model uses to steer its own next-token predictions. The term chain-of-thought is widely used in the industry, but the more conservative term, intermediate tokens, is also used in technical discussions. They discuss how the model’s steps are not always faithful to the actual internal computation. This leads to the crucial distinction between faithfulness (do the tokens accurately reflect the model’s underlying process) and causality (do the tokens causally influence the final output).

Evidence challenging faithfulness

A central portion of the discussion centers on empirical evidence that questions the faithfulness of intermediate tokens. Pavlis notes a growing body of research showing brittleness in these systems and the existence of “failure states” where even correct final outputs can emerge from incorrect or irrelevant reasoning traces. The Illusion of Thinking paper from Apple’s ML team is cited as a watershed viral argument that the model can produce correct outputs even when its supposed chain-of-thought traces are faulty or nonsensical. The podcast references a 2025 survey article titled Failure States in Large Reasoning Models, which catalogued hundreds of cases where the chain-of-thought mechanism did not behave as a faithful, causal chain leading to the right answer.

Decorative tokens and their implications

Another key line of evidence cited is that a significant portion of intermediate tokens may be decorative; removing or altering them does not necessarily derail the model’s final answer. In the most striking experiments, researchers replaced reasoning traces with unrelated tokens (like placeholders or dots) and observed that the model’s performance was not significantly affected. This challenges the intuitive narrative that the chain of thought is the direct mechanism by which the model solves complex tasks. Pavlis and Rao discuss the possibility that the model’s internal dynamics could be more akin to approximate retrieval from a memory-like system rather than true step-by-step deduction.

Two working hypotheses in focus

Subarao Kambapati’s perspective is highlighted as a leading alternative to the traditional chain-of-thought view. He proposes that LRMs are not databases or deterministic calculators but fuzzy memory systems that rely on approximate retrieval to generate outputs. He uses analogies to human memory, such as “mumbling” and “memory jogging,” to describe how self prompts might facilitate better next-token predictions without implying genuine reasoning. This hypothesis aligns with the observation that LRMs excel on verifiable domains like coding and mathematics where outputs can be checked for correctness over multiple steps, while still displaying systematic failure modes elsewhere. The conversation emphasizes that this is not a settled issue, and a synthesis of both views could be necessary to explain why models show both remarkable strengths and surprising weaknesses.

Impact of model openness and interpretability

Another important thread concerns the secretive nature of many models and the resulting challenges for scientific understanding. Pavlis notes that open models provide a window into the model’s inner workings, enabling the research community to observe faithfulness and causality more directly. Closed models, being proprietary and expensive, restrict the ability to inspect reasoning traces and, as a result, complicate interpretability. The discussion acknowledges that the best-performing models currently are often closed, which raises questions about the reproducibility and transparency of results in forward-looking AI research.

Does it matter what we call it?

The conversation turns to whether labeling the process as “reasoning” has real significance. Pavlis emphasizes that language matters because it shapes how researchers think about what the models are doing and what questions they pursue. He invokes the concept of wishful mnemonics, a cautionary view that language can create a cognitive bias toward believing the model is thinking in human-like ways. The speakers advocate for precise but accessible terminology, suggesting that the field may eventually adopt language that describes a spectrum of capabilities from approximation and retrieval to genuine reasoning, rather than a binary label.

Practical implications for science and trust

The episode ends with a discussion about the practical implications for science and the governance of AI in research contexts. If AI tools are to become trusted instruments for scientific discovery, we need robust testing, transparent reporting on model limitations, and careful consideration of the edges of capabilities and the contexts in which the models excel or fail. Pavlis and Patel argue that understanding the limitations and failure modes, even if not fully unveiling the internal mechanisms, is essential to leverage AI responsibly as a scientific instrument.

Conclusion and takeaways

The podcast invites listeners to read Pavlis’s Qualia essay for a deeper examination and to consider how the ongoing research into AI reasoning will shape our understanding of AI as a tool for scientific inquiry. The discussion underscores the importance of precise language, critical evaluation of results, and a cautious but constructive approach to integrating AI into research workflows.