Formula Reference
Every number in your thesis defence — what formula produces it, where each variable comes from, what each variable does.
1 Kit-Build Reconstruction Score
The Formula
S = (M ÷ TGoal) × 100%
| Variable | Meaning | Where it comes from | What it does |
|---|
| M |
Matching propositions |
System compares learner map against goal map. Counts how many propositions (node–link–node triples) the learner placed exactly as in the goal map. |
Numerator — the "correct" part. Higher M = more accurate reconstruction. |
| TGoal |
Total propositions in goal map |
Fixed number set by the teacher when creating the goal map. In this study: 7 propositions representing spatial and functional relationships in わたしのうち. |
Denominator — the max possible. Makes the score a percentage. |
| S |
KB score (%) |
Computed by Yomilink's automatic diagnosis engine. Each proposition is checked: is it in the goal map? If yes → matching. If not → missing or excessive. |
Final output. 60.39% mean = ~4/7 propositions correct on average. |
Result: Mean = 60.39% → avg M = 60.39/100 × 7 ≈ 4.2 propositions
Match/Miss/Excess: The system actually computes three counts for each learner map — matching (correctly placed), missing (in goal map but absent in learner map), and excessive (present in learner map but not in goal map). The score uses only matching. Missing and excessive are diagnostic — they tell the learner what to fix, not just a score.
Concrete Example
Goal map has 7 propositions:
- House → located near → Park
- Library → in front of → Park
- Coffee shop → in front of → Park
- People → borrow books at → Library
- People → read at → Library
- People → drink coffee at → Coffee shop
- Park → is → quiet
Learner correctly reconstructs propositions 1, 2, 6, 7 → M = 4 → S = 4/7 × 100% = 57.1%
Propositions 3, 4, 5 are missing. If the learner also created an extraneous proposition like "Park → has → School," that's excessive — doesn't affect the score.
2 Descriptive Statistics (Mean, Median, SD, Range)
Mean (Average)
x̄ = (x₁ + x₂ + ... + xₙ) ÷ n
| Variable | Source |
|---|
| xᵢ | Each participant's score. For pre-test: extracted from experiment-results.csv column pre_raw, filtered to valid participants (norm_status = "Valid"). For post: post_raw. For delta: post − pre. For KB: kit_score_pct with "%" stripped. |
| n | Number of valid participants = 31 (excludes 4 rushed). From compute-inferential.py: valid_rows filtered by classify_valid(). |
Pre mean: (sum of all 31 pre scores) ÷ 31 = 7.13
Post mean: (sum of all 31 post scores) ÷ 31 = 7.94
Delta mean: (sum of all 31 deltas) ÷ 31 = +0.81
KB mean: (sum of all 31 KB scores) ÷ 31 = 60.39%
Median (Middle Value)
Sort scores ascending → pick middle one (n=31 → 16th value)
| Variable | Source |
|---|
| sorted scores | Same arrays as mean. np.median(arr) in Python; arrayStats.at("50percentile") in Typst via statastic. |
Pre median: 8.00 (half scored ≤ 8, half ≥ 8)
Post median: 9.00
Delta median: 1.00
KB median: 71.00%
Mean vs median: Mean (60.39%) < median (71%) for KB. This means a few very low scores pulled the average down. The distribution is left-skewed — most did reasonably well, a few did poorly.
Standard Deviation (Spread)
s = √[ Σ(xᵢ − x̄)² ÷ (n − 1) ]
| Variable | Source |
|---|
| xᵢ − x̄ | Deviation of each score from the mean. Squared to remove sign, then summed. |
| n − 1 | Sample correction (Bessel's correction). Using n would underestimate true population SD. ddof=1 in numpy std(). |
Pre SD: 2.35 → most scores fell between 4.78 and 9.48
Post SD: 2.10 → slightly tighter after intervention
Delta SD: 2.02 → substantial individual variation in how much people changed
KB SD: ~32% (wide spread — 0% to 100%)
Rule of thumb: In a normal distribution, ~68% of scores fall within ±1 SD of the mean, ~95% within ±2 SD. Pre: 68% of scores roughly between 4.8 and 9.5.
Range (Min, Max)
Min = smallest value, Max = largest value (after sorting)
| Variable | Source |
|---|
| .first() / .last() | Typst: preScoresArr.sorted().first(). Python: min(arr), max(arr). |
Pre: 1 to 10. Post: 2 to 10. Delta: −6 to +4. KB: 0% to 100%.
3 Shapiro–Wilk Normality Test
What it tests
H₀: The data comes from a normal distribution (bell curve).
H₁: The data does not come from a normal distribution.
W = [Σ aᵢ · x₍ᵢ₎]² ÷ Σ (xᵢ − x̄)²
| Variable | Source / Meaning |
|---|
| x₍ᵢ₎ | Ordered scores (sorted from smallest to largest). |
| aᵢ | Weights from Shapiro-Wilk coefficients — pre-computed constants based on sample size. Larger weight for extreme values. |
| W | Ranges 0 to 1. W close to 1 = data is normal. W far from 1 = non-normal. |
| p-value | Probability of seeing W this extreme if data were truly normal. p < 0.05 → reject H₀ → data is non-normal. |
Pre: W = 0.916, p = 0.019 → non-normal
Post: W = 0.847, p = < 0.001 → non-normal
Delta: W = 0.889, p = 0.004 → non-normal
Why non-normal? Bounded scores (0–10) + ceiling effect (many scored 8–10) → distribution is clumped at top, not bell-shaped. This is expected, not broken.
Computed by: scipy.stats.shapiro(arr) in compute-inferential.py line 42–44.
4 Paired t-test (RQ1 Main Test)
The Formula
t = d̄ ÷ (sd ÷ √n)
| Variable | Meaning | Source | Role |
|---|
| d̄ |
Mean of differences (post − pre) |
Sum all 31 deltas, divide by 31. = +0.81. |
Signal — the effect we want to detect. Larger = stronger effect. |
| sd |
Standard deviation of the differences |
SD of the 31 deltas = 2.02. |
Noise — how much people vary in their change. Larger = more noise = harder to detect signal. |
| n |
Number of pairs |
31 valid participants. |
Sample size. Larger n → smaller denominator → larger t → easier to reach significance. |
| √n |
Square root of sample size |
√31 ≈ 5.57 |
Standard error shrinks as n grows. Doubling n doesn't halve error — it reduces by √2. |
| t |
Test statistic |
0.81 ÷ (2.02 ÷ 5.57) = 2.219 |
Signal-to-noise ratio. |t| > ~2.0 → likely significant. |
| p |
Probability under H₀ |
From t-distribution with 30 df. 0.017 (one-sided). |
1.7% chance of seeing |t| ≥ 2.219 if pre and post truly equal. |
t(30) = 2.219, p = 0.017 → statistically significant positive direction.
Breakdown:
Step 1: d̄ = Σ(post − pre) ÷ 31 = 25 ÷ 31 = 0.806
Step 2: sd = √[Σ(dᵢ − 0.806)² ÷ 30] = 2.023
Step 3: SE = 2.023 ÷ √31 = 2.023 ÷ 5.568 = 0.363
Step 4: t = 0.806 ÷ 0.363 = 2.219
Step 5: Look up t=2.219, df=30 in t-table → p ≈ 0.017
Computed by: scipy.stats.ttest_rel(pre, post, alternative="less") in compute-inferential.py line 47. "less" means H₁: pre < post (i.e., scores went up).
5 Cohen's d (Paired Effect Size)
The Formula
d = |d̄| ÷ sd
| Variable | Meaning | Source |
|---|
| d̄ | Mean difference (same as t-test) | +0.81 (from the 31 deltas) |
| sd | SD of differences (same as t-test) | 2.02 (from the 31 deltas) |
d = 0.81 ÷ 2.02 = 0.399
| d value | Interpretation (Cohen, 1988) |
|---|
| 0.2 | Small — effect is subtle, requires large sample to detect |
| 0.5 | Medium — visible to the naked eye |
| 0.8 | Large — obvious, practical importance |
d = 0.399 → small-to-medium. The pre-post shift is about 0.4 standard deviations.
What d actually means: If d = 0.4, the average post-test score is 0.4 standard deviations above the average pre-test score. About 66% of people would score higher at post than pre (if distributions are normal). Think of it as "how much overlap between pre and post distributions" — d = 0 means complete overlap, d = 2 means almost no overlap.
Computed by: mean_diff / std_diff in compute-inferential.py line 50.
6 Wilcoxon Signed-Rank Test (Non-parametric Backup)
What it does (no formula needed)
The Wilcoxon signed-rank test asks the same question as the paired t-test — "did scores change?" — but does not assume normal data.
W = sum of signed ranks of the differences
| Step | What happens |
|---|
| 1 | Compute difference for each person: post − pre. In our data: 31 deltas. |
| 2 | Discard zeros (9 participants had delta = 0). |
| 3 | Rank the absolute values of remaining deltas from smallest (rank 1) to largest. In our data: 22 non-zero deltas. |
| 4 | Assign sign to each rank: positive if delta > 0, negative if delta < 0. |
| 5 | Sum the positive ranks (W⁺) and negative ranks (W⁻). W is the smaller of the two sums. |
W = 52.000, p = 0.014 → confirms paired t-test result.
Intuition: If pre and post were identical, positive and negative ranks would be balanced → W⁺ ≈ W⁻ → large p. In our data, positive ranks dominate → W is small → small p → significant change.
Computed by: scipy.stats.wilcoxon(post, pre, alternative='two-sided') in compute-inferential.py.
7 IQR Outlier Detection
The Formula
IQR = Q3 − Q1
Lower fence = Q1 − 1.5 × IQR
Upper fence = Q3 + 1.5 × IQR
Outlier: any value < lower fence or > upper fence
| Variable | Meaning | Source |
|---|
| Q1 | 25th percentile of deltas | Sort 31 deltas → value at position 8 (25% point). In our data: 0.0. |
| Q3 | 75th percentile of deltas | Sort 31 deltas → value at position 24 (75% point). In our data: 2.0. |
| IQR | Interquartile range — middle 50% spread | 2.0 − 0.0 = 2.00 |
| 1.5 | Conventional multiplier (Tukey, 1977) | 1.5 is standard — captures ~99.3% of normal data. 3.0 is sometimes used for "extreme" outliers. |
| Lower fence | −3.00 | 0 − 1.5×2 = −3 |
| Upper fence | 5.00 | 2 + 1.5×2 = 5 |
1 outlier found: Δ = −6 (below −3). Fences: [−3.00, 5.00].
Trap: The IQR method flags outliers purely mathematically — it doesn't mean the data is wrong. The −6 participant may have genuinely declined. That's why we keep them in the main analysis and only exclude in sensitivity.
Outlier-Excluded Sensitivity
Remove the one outlier → re-run paired t-test with n = 30.
t = 3.520, p < 0.001, d = 0.643 (medium-to-large)
The result holds and actually gets stronger without the outlier. This proves the finding isn't driven by one extreme case.
8 Spearman Rank Correlation (KB Score vs Gain)
The Formula
ρ = 1 − [ 6 × Σ dᵢ² ] ÷ [ n(n² − 1) ]
| Variable | Meaning | Source |
|---|
| dᵢ | Difference in ranks for participant i. Rank their KB score (1=lowest) and their delta (1=lowest), then subtract: rank(KB) − rank(delta). |
From 31 data points: kit_scores array and delta array. |
| n | Sample size | 31 |
| ρ (rho) | Ranges from −1 to +1. ρ = 0 → no monotonic relationship. | 0.198 |
ρ = 0.198, p = 0.284 → not significant. No relationship between how well you reconstructed the concept map and how much your reading score changed.
Why Spearman instead of Pearson? Spearman uses ranks, not raw values. It asks "do people who rank high on KB also rank high on gain?" It doesn't assume a straight-line relationship. With skewed data (KB scores range 0–100%, deltas range −6 to +4), rank-based correlation is more honest.
Computed by: scipy.stats.spearmanr(kit, delta) in compute-inferential.py line 53.
9 Pearson Correlation (Pre-Score vs Gain)
The Formula
r = Σ[(xᵢ − x̄)(yᵢ − ȳ)] ÷ √[Σ(xᵢ − x̄)² × Σ(yᵢ − ȳ)²]
| Variable | Meaning | Source |
|---|
| xᵢ | Pre-test score for participant i | From pre array (31 values). |
| yᵢ | Gain (delta = post − pre) for participant i | From delta array (31 values). |
| x̄, ȳ | Means of pre scores and deltas | 7.13 and 0.81 respectively. |
| Numerator | Covariance — how much x and y move together | Σ (deviation of pre) × (deviation of gain). Negative when high pre → low gain. |
| Denominator | Product of SDs — normalises to [−1, 1] | √[SSpre × SSdelta] |
r = −0.549, p = 0.001 → strong negative correlation.
Pre mean 7.13, gain mean 0.81. Participants with high pre scores (8–10) have deltas near 0 or negative. Participants with low pre scores (1–6) have positive deltas. The negative r quantifies this pattern.
What r = −0.549 means: r² = 0.301 → about 30% of the variance in gain can be "explained" by pre-test score. In plain English: how much room you had to improve predicted about a third of how much you actually improved. The other 70% is other factors.
Computed by: scipy.stats.pearsonr(pre, delta) in compute-inferential.py.
10 Median Split & Welch t-test
Median Split
Median pre = 8.0
Low group: pre ≤ 8 → n = 19
High group: pre > 8 → n = 12
| Variable | Meaning | Source |
|---|
| Median pre | 8.0 — the middle value of the 31 pre-scores | np.median(pre) |
| Low group | 19 participants, mean delta +1.53, SD 1.78 | Filter: pre[i] ≤ 8.0 |
| High group | 12 participants, mean delta −0.33, SD 1.92 | Filter: pre[i] > 8.0 |
Welch Independent t-test
t = (x̄₁ − x̄₂) ÷ √(s₁²/n₁ + s₂²/n₂)
| Variable | Meaning | Source |
|---|
| x̄₁, x̄₂ | Mean delta of low group (+1.53) and high group (−0.33). Difference = 1.86. | From the split data. |
| s₁², s₂² | Variance of each group's deltas: 1.78² = 3.17, 1.92² = 3.69. | Computed per group. |
| n₁, n₂ | 19 and 12. | From the split. |
t = 1.86 ÷ √(3.17/19 + 3.69/12) = 1.86 ÷ √(0.167 + 0.307) = 1.86 ÷ 0.689 = 2.701
Why Welch, not Student's t? Welch's t does not assume equal variances in the two groups. Levene's test confirmed variances are roughly equal (p = 0.270), so using Welch is conservative — it's always safer, never worse.
p = 0.013 → significant difference between low-pre and high-pre gains.
Computed by: scipy.stats.ttest_ind(low_delta, high_delta, equal_var=False) in compute-inferential.py. equal_var=False triggers Welch's correction.
Levene's Test (Variance Equality Check)
Tests H₀: σ₁² = σ₂² (variances equal)
F = 1.264, p = 0.270. p > 0.05 → cannot reject equal variances. Variances are similar enough.
Levene's test is a prerequisite check before choosing Welch vs Student's t. p > 0.05 means we could use Student's t, but Welch is still used because it's always correct.
Mann-Whitney U (Non-parametric Group Comparison)
Ranks all 31 deltas (ignoring group). Sums ranks for each group. U = n₁n₂ + n₁(n₁+1)/2 − R₁ (or the smaller sum).
| Variable | Meaning |
|---|
| R₁ | Sum of ranks for low group after ranking all 31 deltas together. |
| U | Test statistic — measures how separated the two groups' ranks are. |
U = 168.0, p = 0.027 → confirms significant group difference.
Why use both Welch and Mann-Whitney? Same logic as paired t + Wilcoxon — parametric and non-parametric versions of the same question. Both agree → result is robust to distributional assumptions.
11 Hedges' g (Independent Effect Size)
The Formula
g = d × correction
where:
d = (x̄₁ − x̄₂) ÷ spooled
spooled = √[ ((n₁−1)s₁² + (n₂−1)s₂²) ÷ (n₁ + n₂ − 2) ]
correction = 1 − 3 ÷ (4(n₁ + n₂) − 9)
| Variable | Meaning | Source |
|---|
| x̄₁ − x̄₂ | Mean difference: 1.53 − (−0.33) = 1.86 | Group means |
| spooled | Weighted average of group SDs: √[(18×3.17 + 11×3.69) ÷ 29] = √[97.59/29] = √3.365 = 1.834 | Combines both groups' variance, weighted by n−1. |
| d | Cohen's d (independent): 1.86 ÷ 1.834 = 1.015 | Large by Cohen's benchmarks. |
| correction | Hedges' small-sample correction: 1 − 3/(4×31−9) = 1 − 3/115 = 0.9739 | Shrinks d when n is small. At large n, correction → 1. |
| g | Hedges' g: 1.015 × 0.9739 = 0.988 | Large effect. |
g = 0.988 → large effect. The gap between low-pre and high-pre group gains is nearly 1 full standard deviation.
Why Hedges' g instead of Cohen's d? Cohen's d slightly overestimates effect size when samples are small (bias ~3% at n=31). Hedges' g applies a correction factor that removes this bias. For n > 50, d and g are nearly identical. For n=19 and n=12 groups, the correction matters — g (0.988) is more honest than d (1.015).
Computed by: compute-inferential.py lines 117–119: cohen_d_ind / pooled_std then hedges_correction = 1 - 3/(4*N-9) then multiply.
12 TAM Construct Means
The Formula
PU = (PU1 + PU2 + PU3 + PU4 + PU5) ÷ 5
PEoU = (PEoU1 + PEoU2 + PEoU3 + PEoU4 + PEoU5) ÷ 5
| Step | Source |
|---|
| 1. Per-item mean | For each TAM item (e.g., PU1), take average of all 35 participants' responses on the 5-point scale. aggregated.csv columns 24–33. |
| 2. Construct mean | Average the 5 per-item means for PU, separately for PEoU. |
PU = 3.67/5, PEoU = 3.55/5
What "above threshold" means: 3.5 is midpoint between Neutral (3) and Agree (4) on the Likert scale. PU = 3.67 means, on average, participants lean toward "I agree Yomilink is useful." PEoU = 3.55 means they lean toward "I agree it's easy to use." Both are above neutrality but not near "Strongly Agree" (5).
Computed by: tam.typ in paper/report/data/tam.typ — reads aggregated.csv, filters to experiment participants with TAM responses, averages per column.
13 Qualitative Feedback Counts
How they were counted
Three open-ended questions collected from all 35 participants. Researcher read all responses, assigned codes inductively (codes emerged from the data, not predetermined). Codes were grouped into themes. Each response could be assigned multiple themes. Frequency = how many participants mentioned each theme.
| Theme | Count | What it means |
|---|
| fbHelpsLearning | 24 | 24 out of 35 participants said KB helped their learning in some way. |
| fbInitialConfusion | 12 | 12 said they were confused at first about how the task worked. |
| fbMoreVocab | 14 | 14 suggested adding more vocabulary content. |
Not mutually exclusive: One participant can contribute to multiple themes. A person can say "it helped my learning" AND "I was confused at first" AND "add more vocabulary." So the sum of counts exceeds 35.
14 Engagement Classification
How engagement was determined
Based on test completion duration (seconds), recorded in experiment-results.csv as pre_dur_sec and post_dur_sec. Categories:
| Category | Threshold | Meaning |
|---|
| Engaged | Full effort | Completed test at a reasonable pace — reading questions and thinking about answers. |
| Moderate | Partial effort | Some rushing but still mostly engaged. |
| Marginal | Noticeably fast | Likely skimming or guessing. |
| Rushed | < 60 seconds | Implausibly fast for 10 Japanese MCQs. Excluded from analysis. |
Pre-test: 27 engaged, 6 moderate, 0 marginal, 2 rushed.
Post-test: 19 engaged, 9 moderate, 3 marginal, 4 rushed.
Engaged dropped: 77.1% → 54.3%.
Why 60 seconds? 10 items × 6 seconds = 60 seconds. 6 seconds per Japanese MCQ is the absolute minimum to read the question stem and scan options. Anything faster is rapid guessing. This threshold exceeds conventional 2-sec-per-item survey cutoffs to account for Japanese reading load (@Rios2021, @Sideridis2024).
15 Complete Variable Reference
| Value | Formula | Variables used | Data source |
|---|
| 7.13 | Σpre ÷ 31 | 31 pre_raw values | experiment-results.csv, valid only |
| 7.94 | Σpost ÷ 31 | 31 post_raw values | experiment-results.csv, valid only |
| +0.81 | Σ(post − pre) ÷ 31 | 31 delta values | post_raw − pre_raw, valid only |
| 2.35 | √[Σ(pre − 7.13)² ÷ 30] | 31 pre scores, pre mean, n−1 | experiment-results.csv, valid only |
| 60.39% | M ÷ T ÷ 100 | 31 kit_score_pct values | Yomilink automatic diagnosis |
| 0.019 | Shapiro-Wilk(pre) | 31 pre scores | scipy.stats.shapiro |
| 2.219 | 0.81 ÷ (2.02 ÷ √31) | d̄=0.81, sd=2.02, n=31 | scipy.stats.ttest_rel |
| 0.017 | t-dist(df=30, t=2.219) | t=2.219, df=30 | scipy.stats.ttest_rel |
| 0.399 | 0.81 ÷ 2.02 | d̄, sd | mean_diff / std_diff |
| 0.198 | Spearman ρ(kit, delta) | 31 kit scores, 31 deltas | scipy.stats.spearmanr |
| −0.549 | Pearson r(pre, delta) | 31 pre scores, 31 deltas | scipy.stats.pearsonr |
| 2.701 | Welch t(low vs high) | 19 low deltas, 12 high deltas | scipy.stats.ttest_ind(equal_var=False) |
| 0.988 | d × (1 − 3/(4×31−9)) | d=1.015, n=31, correction=0.974 | Compute-inferential.py L117-119 |
| 3.67 | Mean of 5 PU items ÷ 35 | 35×5 TAM responses | aggregated.csv cols 24-28 |
| 3.55 | Mean of 5 PEoU items ÷ 35 | 35×5 TAM responses | aggregated.csv cols 29-33 |