← Research updates

ChatGPT in education: an effect in search of a cause

2025 · Journal of Computer Assisted Learning
With Dragan Gašević, Hendrik Drachsler and Paul Kirschner

Studies and meta-analyses now report that ChatGPT improves learning. We argue that most of these effects cannot yet be interpreted.

An old problem in new clothes

Educational technology research has been here before. Decades of media comparison studies set a new technology against "traditional" teaching and attributed any difference to the technology, although the teaching method changed along with it. Much of the early ChatGPT research repeats this design.

ChatGPT, like other "vanilla" generative AI systems, is a tool and not a method for learning or teaching.

Three things a study must tell us

Drawing on the more mature research on intelligent tutoring systems, we name three requirements for an interpretable effect:

1. The treatment. What exactly did students do with ChatGPT, described precisely enough to replicate?
2. The control group. What did the comparison group do instead, and was the instruction otherwise the same?
3. The outcome. Was learning measured with a valid test after the intervention, and not through self-report or performance on the assisted task itself?

What the audit showed

We audited the 19 comparisons on academic performance, from 18 studies, that entered a recent meta-analysis by Deng et al. (2025).

Only four comparisons met all three requirements. For the others, a positive effect could stem from ChatGPT, from a different teaching method, from a weak control condition, or from an outcome that does not capture learning.

What follows

Observed gains cannot, at this time, be confidently attributed to ChatGPT, and pooled effect sizes may overstate or understate its benefits. We suggest narrowing the question from "Does ChatGPT enhance learning?" to specific hypotheses about specific uses, reporting treatments and controls transparently, and holding back strong policy recommendations until a sturdier body of evidence exists.