A new study coauthored by Human Development Lab researcher Carolina Melo examines the potential and current limitations of using artificial intelligence to observe and evaluate teaching practices in the classroom. Published in Computers & Education: Artificial Intelligence, the research provides new evidence on the reliability and accuracy of large language models (LLMs) for automated classroom observation.
As education systems increasingly explore AI-based tools to support teacher feedback and professional development, understanding how reliably these technologies assess classroom practice has become an important research question. Using the TEACH Primary framework, developed by the World Bank to evaluate teaching quality, the study investigates whether AI-generated classroom observations can produce assessments comparable to those of trained human experts.
The researchers compared 8,618 AI-generated classroom evaluations produced by eight different large language models with consensus-based ratings from certified TEACH observers. Drawing on a dataset of 12 primary school classroom videos, the study assessed both the consistency of AI-generated evaluations and the extent to which they aligned with expert judgments.
The findings suggest that, while AI has considerable potential to support classroom observation, current models still face important limitations. None of the evaluated models consistently produced highly reliable assessments across all dimensions of teaching, and agreement with expert evaluators remained moderate. The study also found that reliability and accuracy did not necessarily go hand in hand: models producing more consistent evaluations were not always those that most closely matched expert ratings.
Beyond measuring reliability and accuracy, the researchers also examined how AI models generate their evaluations. Their analysis suggests that large language models tend to rely primarily on explicit verbal information while overlooking contextual and implicit pedagogical evidence that experienced human observers naturally consider when evaluating classroom practice.
The study contributes to a growing body of international research on the use of artificial intelligence in education. Rather than viewing AI as a replacement for expert evaluators, the findings highlight the importance of rigorously validating AI-assisted observation systems and combining technological innovation with pedagogical expertise and human judgment.
The research was conducted by Carolina Melo, Matías Recabarren, and Javiera de la Maza, with support from the FAIN Innovation Fund at Innovación UANDES. The authors also acknowledge the support and encouragement of Ezequiel Molina at the World Bank throughout the development of the project.
