Lesson 7
Defense Prep
Limitations & Future Work
Turn weaknesses into demonstrations of methodological awareness
Limitations are not weaknesses to hide — they are evidence that you understand your study's boundaries. An examiner who hears honest, specific limitations gains confidence in your work. An examiner who hears vague or defensive responses loses confidence.
The Response Formula
For every limitation:
Acknowledge → Explain → Propose alternative
"Yes, [limitation] is a limitation. The reason was [justification for pilot context]. Future work should [concrete improvement]."
All 8 Limitations (in exam question order)
1. No Control Group
Why it matters: Cannot separate Kit-Build effect from practice effect, material familiarity, or repeated exposure.
This was a feasibility pilot. The goal was to validate workflow and gather initial score direction. A controlled experiment comparing Kit-Build with conventional reading activities is the natural next step.
2. Same Pre-test and Post-test Questions
Why it matters: Post-test scores may be inflated by practice effect (familiarity with the same items).
Using identical questions kept item difficulty constant for a preliminary within-subject comparison. This is supported by related Kit-Build studies (Alkhateeb et al., 2016). Future work should use parallel test forms or equivalent items.
3. Dictionary Access Allowed
Why it matters: Measures assisted comprehension, not unaided reading ability.
The study intentionally measured assisted reading comprehension — how beginners perform with realistic support. This is ecologically valid because real-world beginner reading involves dictionary access. Future studies could add an unaided condition.
4. Reading Material with Furigana Visible During Tests
Why it matters: Script-decoding load was reduced, so the test measures structural understanding more than Kanji recognition.
Furigana was provided because participants had not learned all relevant Kanji characters. Without it, the test would measure Kanji knowledge rather than reading comprehension. This is appropriate for a study targeting beginner learners.
5. Same Material Used in Familiarization and Main Session
Why it matters: Possible learning carryover from Day 2 to Day 3.
Day 2 served as platform familiarization, and the same material ensured consistent task format. The 5-day gap reduced direct recall. Carryover is acknowledged but likely minimal because the task focused on concept map reconstruction, not memorization.
6. High Pre-test Scores (Ceiling Effect)
Why it matters: Participants who scored near the maximum had limited room for improvement.
The negative correlation between pre-test score and gain confirms a ceiling effect. This is a known pattern in educational measurement, not a flaw in the intervention. Future work should use items with a wider difficulty range or target lower-proficiency learners.
7. Sample from Two Classes in One Program
Why it matters: Results may not generalize to other populations or institutions.
This was a convenience sample from the available classes. The pilot's goal was feasibility, not generalizability. Future work should include larger, more diverse samples across different proficiency levels and institutions.
8. TAM Is an Early Signal
Why it matters: Perception data was collected immediately after use in a pilot setting.
TAM scores above 3.5 indicate positive initial acceptance, but this should not be interpreted as proof of adoption. Future work should measure acceptance across multiple sessions and compare with alternative tools.
Future Work Roadmap
Immediate Next Steps
- Controlled experiment with a comparison group (Kit-Build vs. conventional reading activity)
- Parallel test forms for pre-test and post-test
- Larger sample across multiple institutions
Medium-Term Extensions
- Delayed post-test to measure retention (1–2 weeks after intervention)
- Unaided comprehension condition (no dictionary, no furigana)
- Multiple proficiency levels (N5, N4, N3)
- Yomilink vs. original Kit-Build platform comparison
Long-Term Vision
- Collaborative Kit-Build (group concept mapping)
- Integration with classroom LMS
- Adaptive difficulty based on learner performance
- Extension to other languages with complex scripts (Chinese, Korean)
The Meta-Lesson
Never say "I know it's a limitation but..."
Instead: "This is an intentional trade-off for a pilot study. The reason was [X]. Future work should [Y]."
Limited studies are not bad studies. A well-designed pilot that honestly reports limitations and proposes concrete next steps is more valuable than an overclaimed experiment with no clear direction. Your contribution is the platform, the procedure, and the feasibility evidence. The limitations define the research agenda for the next study.