Thoughtfully Applied, Research-Backed AI Improves Learning Outcomes Across Education
Questions about AI’s effectiveness in helping students learn is a central theme as the technology and innovative tools continue to transform the educational landscape. Digital advancements have been cited for years as a way to make learning more accessible and efficient, but for the most part, the question has been whether they can truly enhance outcomes for students on practical and measurable levels.
A recently released study, “Scaling the Doer Effect: A Replication Analysis Using AI-Generated Questions,” conducted by researchers at VitalSource and California State Polytechnic University, sheds light on this. Their findings reveal that AI-generated practice questions, informed by learning science principles, can measurably improve student engagement and academic performance. The results signify an important step in the application of AI in education and prove its ability to replicate the “Doer Effect,” a well-established learning science concept.
Putting the Doer Effect into Practice Boosts Student Success
The Doer Effect emphasizes that students learn better when they actively interact with practice questions while studying content. Research stemming from Carnegie Mellon University’s Open Learning Initiative shows practice activities integrated with reading materials are six times more effective for learning than reading alone, and the benefits increase when assessments are paired with immediate feedback.
To explore whether AI-generated practice questions could match the effectiveness of those crafted by humans, researchers conducted a year-long study in two semesters of a cognitive psychology course at California State Polytechnic University (Cal Poly). Over 200 students participated, and AI-generated questions were embedded alongside textbook content in their digital learning platform.
Changes in the study design across semesters provided valuable insights into the impacts on student engagement and exam scores. During the fall 2022 semester, students had until the end of the term to complete practice questions, which led to last-minute scrambles and lower engagement during the semester. During the spring 2023 semester, practice questions were assigned chapter-by-chapter, with due dates before final exams to promote regular engagement leading up to critical assessments.
Key Study Findings
1. High Engagement Leads to Better Learning
The spring 2023 students completed practice with much greater focus. Only 6.8% of their practice attempts were irrelevant to the content being tested, compared to a staggering 40.3% for the fall 2022 group. This shift ensured students concentrated on material aligned with their exams. Additionally, 22.4% of students even went beyond covered course material by completing practice questions from a new chapter that had been added to the latest edition of a textbook.
2. Improved Academic Performance
Students in spring 2023 received higher average exam scores, with a statistically significant 2% increase. Notably, this gain was most pronounced among lower-performing students, many of whom boosted their grades from a C to a B or higher.
3. AI Successfully Activated the Doer Effect
For the first time, the study confirmed that AI-generated practice questions can drive the Doer Effect. By actively engaging with these questions before exams, students achieved measurable improvements in learning outcomes.
Key Study Findings
1. High Engagement Leads to Better Learning
The spring 2023 students completed practice with much greater focus. Only 6.8% of their practice attempts were irrelevant to the content being tested, compared to a staggering 40.3% for the fall 2022 group. This shift ensured students concentrated on material aligned with their exams. Additionally, 22.4% of students even went beyond covered course material by completing practice questions from a new chapter that had been added to the latest edition of a textbook.
2. Improved Academic Performance
Students in spring 2023 received higher average exam scores, with a statistically significant 2% increase. Notably, this gain was most pronounced among lower-performing students, many of whom boosted their grades from a C to a B or higher.
3. AI Successfully Activated the Doer Effect
For the first time, the study confirmed that AI-generated practice questions can drive the Doer Effect. By actively engaging with these questions before exams, students achieved measurable improvements in learning outcomes.
Findings Demonstrate Significant Potential to Overcome Educational Barriers
The implications of this research are profound. When AI is applied thoughtfully and informed by research, it can transform educational experiences not just for the best students, but for all students. The Bookshelf CoachMe resource goes beyond the standards set by traditional human-generated practice methods and actually scales them to reach more students. This development can help to address key educational challenges, such as:
- Time and cost efficiency: AI reduces the labor-intensive process of manual question creation, enabling institutions to offer high-quality practice without added costs and efforts.
- Supporting diverse learners: By strengthening engagement and results for lower-performing students, AI tools help level the playing field and promote equity in education.
Harnessing AI’s Potential Will Shape the Future of Learning
While tools like ChatGPT dominate headlines, this study highlights the importance of pairing AI with a strong foundation in learning science. Unlike generative AI models, which may provide factual inaccuracies or unreliable content, VitalSource’s automatic question generation system creates contextually relevant practice questions directly tied to course content. The purpose-built learning tools developed through this approach inspire trust, improve outcomes, and offer greater consistency.
Looking ahead, continued collaboration between EdTech developers and educators will be critical to unlocking AI’s full potential. Educational technology should never replace the role of instructors but instead can support and empower them to enhance their teaching techniques. When implemented strategically and with care, AI becomes an ally to both educators and learners.
This content was paid for and created by VitalSource. The editorial staff of The Chronicle had no role in its preparation. Find out more about paid content.
Built with Shorthand