Research basis
The learning science behind Math Magic
Every major feature in Math Magic is built on peer-reviewed research about how students learn math. This page explains what we're trying to change, how the product is meant to get there, and the studies behind each design decision. It's written for teachers, families, school leaders, and researchers.
Logic model
How we expect Math Magic to make a difference, from the problem we're solving to the outcomes we hope to see. Each step lists the research that supports it.
Problem statement
Many middle school students reach grade-level math with unfinished learning in earlier prerequisite skills. On the 2022 NAEP, only about a quarter of U.S. eighth graders scored at or above Proficient in mathematics. Whole-class instruction rarely has time to find and repair each student's specific gaps, and teachers lack timely, skill-level data to decide whom to reteach and what to reteach.
National Center for Education Statistics 2022Bloom 1984Inputs
- Standards-aligned diagnostic exams and a tagged question bank for grades 6–8 (CCSS).
- A prerequisite skill map linking each grade-level skill to the earlier skills it depends on.
- An AI math coach (Cyphera) constrained to Socratic questioning, with an offline fallback.
- Worked examples, hints aimed at specific misconceptions, and answer checking that accepts equivalent forms.
- Teacher dashboards, analytics, and reports; class rosters and privacy safeguards (FERPA/COPPA).
- Teacher time: assigning work, reviewing data, and running small-group reteaching.
Activities and strategies
- Students take a diagnostic; each missed problem triggers a remediation loop on the missing prerequisite skill before they move on, ending in an equivalent-rigor mastery check.Bloom 1984Kulik et al. 1990
- When stuck, students work with a coach that asks guiding questions and prompts them to explain their reasoning instead of giving answers.Chi et al. 1994Chi & Wylie 2014VanLehn 2011
- Students study step-by-step worked examples, including common incorrect examples, before and during practice.Sweller & Cooper 1985Atkinson et al. 2000Booth et al. 2013
- Students complete spaced, mixed retrieval practice (Spell Trials, Cauldron sets, games) with fresh numbers on every attempt and retakes.Roediger & Karpicke 2006Cepeda et al. 2006Rohrer et al. 2015
- Every answer gets immediate feedback with a hint aimed at the likely mistake and a second try.Hattie & Timperley 2007Shute 2008
- Teachers review skill-level analytics, early warnings, and suggested groups, then reteach and assign targeted practice.Black & Wiliam 1998Carlson et al. 2011
Outputs
- Diagnostics and mastery checks completed per student.
- Prerequisite gaps identified and remediation loops completed.
- Practice problems attempted, retakes taken, and minutes of practice.
- Coach conversations and worked examples viewed.
- Teacher actions: assignments created, reteach groups formed, reports shared.
Short-term outcomes
- Students close identified prerequisite gaps and score higher on the mastery check than on the diagnostic.Kulik et al. 1990Kulik & Fletcher 2016
- Students explain their reasoning more often and rely less on being told answers.Chi et al. 1994Rittle-Johnson 2006
- Teachers reteach sooner and to the right students, based on skill-level data.Black & Wiliam 1998Carlson et al. 2011
Long-term outcomes
- Higher grade-level math achievement and retention of skills over time.Roediger & Karpicke 2006Cepeda et al. 2006Kulik & Fletcher 2016
- Greater math confidence and persistence, and sustained engagement in practice.Hattie & Timperley 2007Sailer & Homner 2020
- A school culture of data-informed, responsive math instruction.Black & Wiliam 1998Carlson et al. 2011
Key design decisions and the research behind them
Seven distinct choices we made in building Math Magic, what each looks like in the product, and annotated research showing why.
- 1
Diagnose, then repair the missing prerequisite before moving on
In the product
Each diagnostic problem is linked to the earlier skills it depends on. A wrong answer starts a remediation loop on the specific missing skill, and the student moves on only after showing they've got it. The unit ends with an equivalent-rigor mastery check and a before/after report.
Why
Mastery learning, which pairs formative checks with corrective instruction until students reach a set standard, consistently improves achievement compared with conventional instruction.
Annotated research
Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16.
Reports that students tutored one-to-one with mastery learning outperformed conventionally taught peers by about two standard deviations, and frames the search for group methods (like formative tests plus correctives) that approach that effect. Basis for building corrective loops into every diagnostic.
Kulik, C.-L. C., Kulik, J. A., & Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60(2), 265–299.
Meta-analysis of 108 controlled evaluations of mastery learning programs finding positive effects on end-of-course achievement, with the largest gains for lower-achieving students. Supports requiring demonstrated mastery of each prerequisite before advancing.
- 2
The coach asks questions instead of giving answers
In the product
Cyphera, the AI coach, is instructed never to give the final answer to the current problem. It asks one guiding question at a time, prompts students to explain their steps, and stays on math.
Why
Learning improves when students generate explanations and engage constructively or interactively, rather than passively receiving answers. Step-based tutoring that responds to student reasoning approaches the effectiveness of human tutoring.
Annotated research
Chi, M. T. H., De Leeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439–477.
Eighth graders prompted to self-explain while studying a text showed greater understanding than an unprompted control group, and high self-explainers built more accurate mental models. Basis for the coach's prompts to explain each step.
Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243.
Synthesizes evidence that interactive and constructive engagement produce more learning than active or passive engagement (the ICAP framework). Supports a dialogue-based coach over an answer-giving one.
VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.
Review comparing human tutoring with computer tutoring systems found step-based tutoring systems nearly as effective as human tutors (effect size about 0.76 versus 0.79). Supports tutoring at the level of each step rather than only checking final answers.
- 3
Worked examples, including common wrong ways
In the product
When a student misses a problem, they see a step-by-step worked example of a matched problem. Hints and examples name common mistakes, such as sign errors, so students can recognize and avoid them.
Why
For novices, studying worked examples reduces cognitive load and produces better learning than solving equivalent problems unaided. Studying incorrect examples alongside correct ones helps students recognize and correct misconceptions.
Annotated research
Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59–89.
Experiments with algebra students showed that studying worked examples led to faster learning and fewer errors on similar problems than conventional problem solving. Basis for showing worked examples in the remediation loop.
Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181–214.
Review of worked-example research identifying design principles (such as integrating steps and pairing examples with practice). Informs how examples are structured and sequenced.
Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction, 25, 24–34.
Classroom study in algebra found that incorrect worked examples, alone or combined with correct ones, improved conceptual understanding and reduced misconceptions. Basis for hints that name common mistakes.
- 4
Spaced, mixed retrieval practice with fresh numbers and retakes
In the product
Spell Trials, Cauldron sets, and games ask students to recall and apply skills repeatedly over time. Questions are generated with new numbers each attempt, practice mixes problem types, and students can take a Second Casting (retake) on a new version.
Why
Retrieving information through testing strengthens long-term retention more than restudying, spacing practice over time improves retention, and mixing problem types improves students' ability to choose the right strategy.
Annotated research
Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255.
Experiments showed that taking practice tests produced substantially better retention after a delay than restudying the same material. Basis for frequent low-stakes quizzes and retakes.
Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380.
Quantitative synthesis of 317 experiments finding that spaced practice yields better retention than massed practice. Supports spreading practice of each skill across days and weeks.
Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107(3), 900–908.
Randomized classroom study with seventh graders found interleaved math practice produced much higher scores on a delayed test than blocked practice. Supports mixing problem types within practice sets.
- 5
Immediate feedback that targets the mistake, plus a second try
In the product
Answers are checked instantly and accept any correct equivalent form. A wrong answer gets a hint aimed at the likely misconception and another try for partial credit (the Second Chance Charm), rather than only a right/wrong mark.
Why
Feedback is among the most powerful influences on learning when it tells students where they are, where they are going, and what to do next. Elaborated, specific formative feedback is more effective than verification alone.
Annotated research
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.
Synthesis of meta-analyses showing feedback has a large average effect on achievement, strongest when it addresses the task and the process for improving. Basis for hints that tell students what to try next.
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
Review of formative feedback research recommending specific, elaborated feedback delivered in manageable units. Supports targeted hints over simple correct/incorrect marks.
- 6
Skill-level data that tells teachers whom to reteach and what to reteach
In the product
The teacher dashboard, Sigil Map, and Crystal Ball Analytics show mastery by skill and standard, early warnings with reasons, question analysis with the most common gaps, and suggested small groups with one-tap reteaching assignments.
Why
Formative assessment, where teachers use evidence of learning to adjust instruction, raises achievement. Structured use of student data to guide instruction has improved math achievement in a large randomized trial.
Annotated research
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74.
Review of about 250 studies concluding that strengthening formative assessment produces significant learning gains, especially for low achievers. Basis for giving teachers timely, skill-level evidence.
Carlson, D., Borman, G. D., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33(3), 378–398.
Cluster randomized trial across more than 500 schools in seven states found that a district-level data-driven reform improved mathematics achievement. Supports building analytics that lead directly to instructional action.
- 7
Game elements that sustain practice
In the product
Magic-themed games (Arcana Matrix, Spellbound Showdown), points, and a class economy reward effort and accuracy during practice.
Why
Gamification has small-to-moderate positive effects on cognitive, motivational, and behavioral learning outcomes, especially when it includes game fiction and social interaction.
Annotated research
Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32(1), 77–112.
Meta-analysis of gamification in education finding significant positive effects on cognitive, motivational, and behavioral outcomes, with game fiction and combined competition and collaboration as helpful features. Basis for the magic theme and team games.
Our qualitative study
Teacher and student experiences with Math Magic: a qualitative study
A qualitative descriptive study using semi-structured teacher interviews, student focus groups, and open-ended survey responses, analyzed with reflexive thematic analysis.
Research questions
- How do middle school math teachers describe the ways Math Magic informs their instructional decisions?
- How do students describe their experience getting unstuck with Math Magic's coach, worked examples, and feedback?
- What do teachers and students see as the benefits, barriers, and needed improvements of Math Magic?
Findings will be posted here when the study is complete.
Bibliography
- Atkinson, R. K., Derry, S. J., Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of Educational Research, 70(2), 181–214. https://doi.org/10.3102/00346543070002181
- Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
- Bloom, B. S. (1984). The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13(6), 4–16. https://doi.org/10.3102/0013189X013006004
- Booth, J. L., Lange, K. E., Koedinger, K. R., & Newton, K. J. (2013). Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction, 25, 24–34. https://doi.org/10.1016/j.learninstruc.2012.11.002
- Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
- Carlson, D., Borman, G. D., & Robinson, M. (2011). A multistate district-level cluster randomized trial of the impact of data-driven reform on reading and mathematics achievement. Educational Evaluation and Policy Analysis, 33(3), 378–398. https://doi.org/10.3102/0162373711412765
- Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T., & Rohrer, D. (2006). Distributed practice in verbal recall tasks: A review and quantitative synthesis. Psychological Bulletin, 132(3), 354–380. https://doi.org/10.1037/0033-2909.132.3.354
- Chi, M. T. H., & Wylie, R. (2014). The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 49(4), 219–243. https://doi.org/10.1080/00461520.2014.965823
- Chi, M. T. H., De Leeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1207/s15516709cog1803_3
- Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
- Kulik, C.-L. C., Kulik, J. A., & Bangert-Drowns, R. L. (1990). Effectiveness of mastery learning programs: A meta-analysis. Review of Educational Research, 60(2), 265–299. https://doi.org/10.3102/00346543060002265
- Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: A meta-analytic review. Review of Educational Research, 86(1), 42–78. https://doi.org/10.3102/0034654315581420
- National Center for Education Statistics. (2022). The Nation's Report Card: 2022 mathematics assessment results at grades 4 and 8. U.S. Department of Education, Institute of Education Sciences. https://www.nationsreportcard.gov/highlights/mathematics/2022/
- Rittle-Johnson, B. (2006). Promoting transfer: Effects of self-explanation and direct instruction. Child Development, 77(1), 1–15. https://doi.org/10.1111/j.1467-8624.2006.00852.x
- Roediger, H. L., & Karpicke, J. D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17(3), 249–255. https://doi.org/10.1111/j.1467-9280.2006.01693.x
- Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107(3), 900–908. https://doi.org/10.1037/edu0000001
- Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32(1), 77–112. https://doi.org/10.1007/s10648-019-09498-w
- Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
- Sweller, J., & Cooper, G. A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and Instruction, 2(1), 59–89. https://doi.org/10.1207/s1532690xci0201_3
- VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221. https://doi.org/10.1080/00461520.2011.611369
Student privacy
Math Magic operates as a school official under FERPA, follows COPPA's school-authorization rules, never sells student data or shows ads, and doesn't send student names to AI models.
