The Allen Institute for AI (Ai2) presents the TutorMoments framework for measuring large language models’ ability to make one of education’s most difficult decisions: should they provide a student with additional support so they can begin solving a problem, or should they step back and encourage the student to do more of the thinking independently? The framework relies on replaying real moments from one-on-one math tutoring sessions rather than merely using tests that reward a single behavior, such as always providing a hint or always refraining from revealing the answer.
A preview of the project was published in a community article on the Hugging Face blog on August 7, 2026, written by Kyle Wiggers on behalf of Ai2Comms. The announcement includes a technical report, a dataset available on Hugging Face, code for running the reevaluation pipeline, and session replays generated by the models for the moments being evaluated.
Why Isn’t an Always-Helpful Response Enough?
The project starts from an educational observation: good teaching does not mean performing the hardest part of a task on behalf of the student. A teacher may ask a question that helps diagnose what the student understands, or give the student space to explain a correct answer and demonstrate their understanding. At other times, a student may need the problem to be made more accessible so they can get started.
However, language models are trained to be helpful, which may lead them to explain the concept, organize the steps, and guide the student toward the answer quickly. According to the article, this behavior can shorten what is known as productive struggle—the difficult or sometimes frustrating effort involved in solving a problem, which learning research links to stronger understanding. Therefore, TutorMoments’ developers believe the right question is not whether the model provided a hint or refrained from answering in general, but whether its choice was appropriate for what this student needed at that moment.
How Does TutorMoments Work?
The TutorMoments-Preview version consists of 462 de-identified text conversations from real one-on-one math tutoring sessions with American students in grades two through seven. The data includes more than 1,500 key moments annotated by teachers and several thousand free-form text comments provided by 27 American teachers.
The conversations came from an intensive tutoring program, and most students attended Title I-supported schools. The data was made available under a research provision approved by parents and guardians. Identifying details were first removed by the organization that provided the data, and then through an additional processing pipeline that accounted for math content.
Experienced teachers read the transcripts and identified moments when the tutor had to balance scaffolding, meaning making the problem more accessible, with pushing for rigor, meaning encouraging the student to do more difficult thinking. The system pauses the conversation at one of these points, then hands the tutor’s role to a language model in a five-round simulated session, while another language model plays the student.
A language-model-based scoring pipeline then evaluates three dimensions: whether the model provided scaffolding when the student needed it, whether it pushed the student toward greater rigor when the student was ready, and whether it avoided excessive scaffolding that lowered the level of challenge more than the moment required. The evaluation begins with a reference judgment made by teachers. When they disagreed, the majority label was used; if two out of three believed the moment required rigor, that judgment was adopted as the reference result.
What Did the Initial Results Show?
The team tested seven language models using two types of instructions. The first asks the model to teach well without actually explaining the required balance, while the second explicitly clarifies the difference between scaffolding, over-scaffolding, and pushing the student toward rigor. The moments were divided equally between cases in which scaffolding was the correct choice and cases that required a greater push toward thinking.
The results showed that all models scored better when using instructions that explained the tradeoff. This suggests that the default “helpful assistant” behavior alone is not sufficient for providing good instruction. However, describing the tradeoff in the instructions did not solve the problem completely; the models still differed substantially in how reliably they made the appropriate decision, and even the best models still had clear room for improvement.
The team also found that, when instructed to push the student toward rigor, the models tended to use less varied strategies than teachers and often relied on asking the student to explain their answer. Human teachers, by contrast, used more diverse approaches and were more likely to step back and let the student work independently.
Limits of the Comparison and What It Means
TutorMoments’ results should not be read as evidence that human teachers are less capable than AI. When evaluated using the same method, the human teachers in the data received scores of 0.458 for appropriate scaffolding, 0.182 for appropriate rigor, and 0.496 for avoiding over-scaffolding. These scores were lower than the models’ scores when using evaluation-aware instructions and close to the range of their scores with ordinary instructions.
However, the collection was originally designed around moments that teachers believed could have been taught better, so it focuses on missed opportunities rather than ideal teaching. In addition, the students in the replays are simulated by a language model, meaning that the scores measure the model’s behavior at a decision point and do not measure whether a real student actually learned.
The data also remains geographically and educationally limited: it focuses on math in elementary and middle school within the United States and relied on a single group of educators. The source also notes that detecting when a student should be pushed toward rigor is less reliable than detecting scaffolding, and that there were 260 rigor moments compared with 738 scaffolding moments.
In its current form, TutorMoments offers a more specific way to examine the educational decisions made by models, but it is not a substitute for studies involving real students and measuring learning outcomes. The team plans to collect feedback on the preview before developing a larger, multimodal dataset, a stronger evaluation pipeline, and deeper analysis.