6 Minutes
Since ChatGPT emerged in late 2022, millions have begun using large language models to answer questions, synthesize information, and speed tasks that once required time-consuming searches. A new set of experiments suggests this convenience can come with a cost: people who rely on LLM summaries tend to develop shallower knowledge than those who research via traditional web searches.
How the experiments were run and what they found
Researchers Shiri Melumad and Jin Ho Yun, both marketing professors, led seven controlled studies with over 10,000 participants to test how learning differs when people use an LLM like ChatGPT compared with a standard Google search. Participants were asked to learn about everyday topics, for example how to grow a vegetable garden, and were randomly assigned to either use an LLM or to navigate web links the "old-fashioned way." There were no limits on how long they could search or how many prompts they could give the LLM.
After researching, participants wrote advice to a friend based on what they had learned. Across studies, a clear pattern emerged: those who used LLM summaries reported feeling that they had learned less, invested less effort when composing their advice, and produced shorter, more generic, and less factual write-ups than participants who had dug through web pages themselves.
When independent readers—unaware of which tool was used—evaluated the advice, they consistently rated the LLM-derived guidance as less informative and less helpful, and were less likely to follow it. The findings were robust across multiple topics and population samples.
Using Google 'the old-fashioned way' involves reading more broadly.
Why synthesized answers can weaken deep learning
At the heart of these results is a well-established principle from learning science: active engagement builds stronger, more transferable knowledge. Traditional web searches force learners to read diverse sources, evaluate credibility, interpret differing viewpoints, and synthesize those elements into their own mental representation. That effortful processing creates "desirable difficulties"—a friction that supports durable learning.
LLMs, by contrast, distill many sources into a polished summary, effectively doing the interpretation and synthesis for the user. The interaction becomes more passive: ask a question, receive a coherent answer, move on. The convenience is undeniable, but the trade-off appears to be less deep encoding of facts and fewer mental connections that support flexible use of knowledge later.
The research team tested alternative explanations. One hypothesis was that LLMs simply expose users to a narrower set of facts, producing less eclectic output. To control for that, some experiments presented identical factual content to both LLM and web-search groups. Other studies kept the platform constant—using Google's standard search results versus an AI Overview feature—and still found that synthesized summaries led to shallower understanding than active compilation from web links.
Want a more in-depth understanding? LLMs might not be the right approach.
Practical implications for learners, educators, and professionals
These findings do not imply that LLMs should be banned. Large language models are powerful productivity tools: quick explanations, coding help, drafting, and brainstorming are obvious use cases. The key message is nuance—learners and workers should be strategic in choosing when to lean on an AI co-pilot and when to embrace more effortful research.
For routine factual lookups or when speed is essential, an LLM provides a useful first pass. But when the goal is to develop deep, generalizable knowledge—mastering a subject well enough to teach it, argue about it, or apply it in novel situations—relying solely on synthesized answers is likely to undercut that aim.
The researchers also tested whether mixing the approaches would help. In one experiment, participants interacted with a specialized GPT that provided real-time web links alongside its summary. Even with links available, once users received the polished synthesis they tended not to explore the original sources, and knowledge remained shallower than for those who had navigated links from the outset.
Expert Insight
Dr. Elena Morales, a cognitive scientist specializing in learning and technology, notes: 'Automatic summaries save time but can short-circuit the mental work needed for durable learning. The trick for educators and tool designers is to build healthy frictions—structured prompts, retrieval practice, or required engagement with primary sources—that nudge users to think actively rather than passively accept an answer.'
What could improve LLM-supported learning?
Melumad and Yun suggest a research program focused on design interventions that preserve the efficiency of generative AI while reintroducing productive friction. Possible approaches include tools that prompt learners to explain answers in their own words, require retrieval attempts before providing a summary, or highlight diverse source links and ask targeted follow-up questions that encourage verification and deeper exploration.
Such guardrails could be particularly important in secondary and higher education where building foundational reading, writing, and reasoning skills remains essential. Teachers might pair LLM outputs with assignments that force students to cite original sources, defend claims, or extend summaries into applied projects.
Conclusion
The rise of ChatGPT and other LLMs marks a profound shift in how people access knowledge. Experimental evidence now shows that convenience can come at the expense of depth. Rather than abandoning these tools, the priority should be smarter use: match the tool to the task, and design learning experiences that restore the active processing necessary for deep understanding.
Source: sciencealert
Comments
Reza
is this even true? if that's real then maybe it depends on how ppl use it. I ask followups, test answers, still learn - but others probably skim. where's the age breakdown?
mechbyte
wow didnt expect this. been using ChatGPT for quick answers, and yeah - my memory's fuzzy later. gonna force myself to open links and read more honestly
Leave a Comment