Teacher and student experiences with Math Magic: a qualitative study
Last updated October 28, 2026
This is the full plan for our qualitative study, including research questions, methods, interview and focus group guides, analysis plan, and consent forms. Bracketed fields should be filled in by the research lead before use. Print or save this page as a PDF to share with your district.
1. Purpose and research questions
This study explores how middle school teachers and students experience Math Magic and what benefits, barriers, and improvements they see. It is designed to understand experiences, not to measure effectiveness.
- How do middle school math teachers describe the ways Math Magic informs their instructional decisions?
- How do students describe their experience getting unstuck with Math Magic's coach, worked examples, and feedback?
- What do teachers and students see as the benefits, barriers, and needed improvements of Math Magic?
2. Study design
A qualitative descriptive study using semi-structured teacher interviews, student focus groups, and open-ended survey responses, analyzed with reflexive thematic analysis. Data sources are triangulated: teacher interviews, student focus groups, and anonymous open-ended survey responses collected in the app.
3. Setting and participants
- Setting: one or more middle schools where Math Magic is used in grade 6–8 math classes for at least four weeks before data collection.
- Teachers: 3–6 math teachers who have used Math Magic with at least one class. Purposive sampling for a range of grade levels and years of experience.
- Students: 12–30 students in grades 6–8 from participating classes, grouped into focus groups of 3–5. Purposive sampling across achievement levels, including students with IEPs/504 plans and multilingual learners when families consent.
- Inclusion: currently using Math Magic; for students, a signed parent/guardian permission form and student assent.
- Sample size will be guided by information power; recruitment stops when new interviews add little new information to the research questions.
4. Approvals, consent, and voluntary participation
- Obtain written approval from the school district's research office (or principal, per district policy) before recruiting. If a university partner co-leads, obtain IRB approval or an exemption determination.
- Teachers sign the teacher consent form below. Parents/guardians sign the permission form; students give verbal or written assent.
- Participation is voluntary and has no effect on grades, evaluations, or access to Math Magic. Anyone may skip a question or withdraw at any time.
- Students not participating continue normal class activities. Focus groups take place at a time arranged with the teacher so no instruction is missed.
- In-app surveys are anonymous: no names or student IDs are stored with responses. Students only complete them if their family returned the permission form.
5. Data collection timeline
- Weeks 1–2: approvals, recruitment, consent and permission forms.
- Weeks 1–6: classes use Math Magic as part of normal instruction (at least four weeks).
- Weeks 5–6: anonymous in-app surveys open for teachers and consented students.
- Weeks 6–7: one 30–45 minute teacher interview each; one 25–30 minute focus group per student group. Sessions are audio-recorded with permission.
- Weeks 8–11: transcription, de-identification, and analysis.
- Week 12: member checking with teachers, then the final report.
6. Teacher interview guide (semi-structured)
Follow-up probes: “Can you tell me more?” “Can you give an example?” “How did students respond?”
- Tell me about your classes and how you've been using Math Magic.
- Walk me through a recent time you looked at Math Magic data. What did you notice, and what did you do next?
- How, if at all, has Math Magic changed how you group students or decide what to reteach?
- What have you noticed about how students use the coach, worked examples, and retakes when they're stuck?
- Which students seem to benefit most? Are there students for whom it works less well? Why do you think that is?
- What gets in the way of using Math Magic the way you'd like (time, devices, setup, content)?
- How does Math Magic fit with your curriculum and other tools?
- If you could change one thing about Math Magic, what would it be?
- Is there anything else you'd like to share?
7. Student focus group guide
Opening script: “Thanks for helping us. There are no right or wrong answers, and nothing you say affects your grade. We want to know what Math Magic is really like for you, including what you don't like. You can skip any question or stop at any time. Please use first names only and keep what others say here private.”
- When you open Math Magic, what do you usually do?
- Tell us about a time you got stuck on a problem. What happened next?
- What is it like when the coach asks you questions instead of telling you the answer?
- How do the worked examples or hints help you, or not help you?
- How do you feel about getting another try or a retake?
- Has using Math Magic changed how you feel about math? How?
- What is confusing or frustrating?
- If you were in charge of Math Magic, what would you change?
8. Analysis plan
Transcripts and open-ended survey responses will be analyzed using reflexive thematic analysis (Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.).
- Phase 1: become familiar with the data by reading all transcripts and responses and taking notes.
- Phase 2: generate initial codes, both inductive and tied to the logic model (for example “uses data to regroup”, “coach prompts reasoning”).
- Phase 3: build candidate themes by grouping codes.
- Phase 4: review themes against the coded extracts and the full dataset.
- Phase 5: define and name themes with clear descriptions.
- Phase 6: write up themes with representative de-identified quotations, reported separately for teachers and students.
- Two analysts code independently and meet to discuss differences until they agree. A shared codebook and decision log provide an audit trail.
9. Trustworthiness
- Triangulation across teachers, students, and surveys.
- Member checking: teachers review a summary of themes for accuracy.
- Reflexivity: because the product team has a stake in the results, an independent researcher or educator (not employed by Math Magic) will run sessions and co-code wherever possible. Analysts keep reflexive memos.
- Disconfirming evidence and negative experiences are reported, not just positive ones.
- Thick description of the setting so readers can judge whether the findings apply elsewhere.
10. Data protection
- Participants are given pseudonyms in transcripts and reports. A key linking names to pseudonyms is stored separately and destroyed at the end of the study.
- Audio recordings are stored encrypted, accessible only to the research team, and deleted after transcripts are verified.
- Study data is kept separate from student product data and is never used for marketing with identifiable information.
- Only aggregate findings and de-identified quotations are published.
11. Limitations
- Small, purposive sample in a limited number of schools; findings describe experiences and are not meant to generalize statistically.
- Possible social-desirability bias, reduced by independent facilitators, anonymous surveys, and explicitly inviting criticism.
- This study does not measure causal effects on achievement.
Form A: Teacher consent
You are invited to take part in a study about teachers' and students' experiences with Math Magic. You will be asked to take part in one 30–45 minute interview, which will be audio-recorded with your permission, and you may complete a short anonymous survey. Participation is voluntary and will not affect your employment, evaluation, or access to Math Magic. You may skip questions or withdraw at any time. Your name will not appear in any report.
Questions? Contact: [Research lead name, email, phone].
- I agree to take part in this study. ☐ Yes ☐ No
- I agree to have my interview audio-recorded. ☐ Yes ☐ No
- Printed name: ______________________________
- Signature: ______________________________ Date: ______________
Form B: Parent/guardian permission
Your child's math class uses Math Magic, an online math practice program. We are doing a study to learn what students think about it. With your permission, your child may join one small-group conversation (25–30 minutes, 3–5 students) during a time that does not take away from instruction, and may answer a short anonymous survey in the app. The conversation will be audio-recorded so we can remember what was said; recordings are deleted after they are typed up. Your child's name will never appear in any report.
Taking part is completely voluntary. It will not affect your child's grades. Your child may stop at any time, and you may withdraw permission at any time by contacting the teacher. Students who don't take part continue normal class activities.
Questions? Contact: [Research lead name, email, phone] or your child's teacher.
- ☐ Yes, my child may take part. ☐ No, my child may not take part.
- ☐ I agree to audio recording of the group conversation.
- Child's name: ______________________________ Teacher/period: ______________
- Parent/guardian name: ______________________________
- Signature: ______________________________ Date: ______________
Form C: Student assent script (read aloud)
“We're trying to learn what students think about Math Magic so we can make it better. If you want to help, you'll talk with a few classmates and me for about 25 minutes. I'll record us so I don't forget what you said, but I won't use your name when I write about it. There are no right or wrong answers, and it won't change your grade. You can skip any question or stop any time, and nobody will be upset. Your family already said it's okay, but it's your choice too. Do you want to take part?”
- Student agreed: ☐ Yes ☐ No Facilitator initials: ______ Date: ______________
