Why Each Test Was Chosen

Every statistical decision grounded in your data characteristics, research design, and the literature cited in your paper.

1 The Decision Tree

Here is how each test connects to the next — a chain of decisions forced by your data:

2 Shapiro–Wilk (Normality Test)

Step	Question	Answer	Forced decision
1	Is the data normal?	No — all three distributions fail Shapiro-Wilk (ceiling effect, bounded scores)	Must check normality first → Shapiro-Wilk
2	Then can we use a t-test?	Yes, despite non-normality — CLT robust at n ≥ 30 (n=31)	Paired t-test as primary, but needs non-parametric backup → Wilcoxon
3	Is t-test enough to report?	No — p-value alone doesn't tell you how big the effect is	Must report effect size → Cohen's d
4	Are there outliers distorting the result?	Yes — IQR detects one outlier at Δ = −6	Must do sensitivity analysis → re-run t-test without outlier
5	Does KB score predict reading gain?	Need rank-based correlation (non-normal data)	Spearman ρ, not Pearson (for this pair)
6	Does pre-test score predict gain?	Theory predicts linear negative relationship	Pearson r (straight-line test for ceiling effect)
7	Is the ceiling effect real?	Need to split sample and compare groups	Median split → Welch t + Mann-Whitney + Hedges' g

What question does it answer?

"Can I assume my data follows a bell curve, or do I need to be careful about which tests I choose?"

Why this test, in your specific study?

Mandatory prerequisite. In classical hypothesis testing (Neyman-Pearson framework, which your paper follows), normality is an assumption of the paired t-test. You must check it before reporting the t-test — otherwise an examiner will ask "did you even check?" Running Shapiro-Wilk is the standard way to answer that question transparently.

Bounded scores produce predictable non-normality. Your pre and post scores are bounded 0–10. With 51.6% of participants scoring ≥ 8 and 16.1% scoring a perfect 10, the distribution piles up against the upper boundary. This is a textbook case where Shapiro-Wilk will reject normality — which is expected, not alarming. Running it and reporting it shows you understand your data's structure.

Shapiro-Wilk specifically, not Kolmogorov-Smirnov. Shapiro-Wilk has higher statistical power for small-to-moderate samples (n < 50). Your n = 31 makes Shapiro-Wilk the right choice. K-S is less sensitive at this sample size and would risk a false "normal" verdict when the data is actually non-normal.

Kolmogorov-Smirnov — lower power at n < 50, more likely to miss non-normality. Anderson-Darling — designed for continuous distributions, less common in social science.

Cited in paper: @LiuWang2021 — establishes that bounded score data with ceiling effects naturally produces non-normal distributions, which is expected rather than problematic.

3 Paired t-test (RQ1 Primary)

What question does it answer?

"Given that the same 31 people took both tests, is the average change from pre to post large enough that it probably isn't random noise?"

Why this test, in your specific study?

Matched the design. You have a one-group pretest-posttest design — the same 31 people measured twice. A paired test is the only correct choice because it controls for individual differences. An independent t-test would treat the pre and post scores as if they came from different people, which loses the pairing information and reduces power.

CLT rescue. Shapiro-Wilk says your data is non-normal (ceiling effect, bounded scores). But the Central Limit Theorem kicks in at n ≥ 30: the sampling distribution of the mean difference approaches normality even when raw scores don't. At n = 31 (just barely clearing the threshold), the t-test is considered robust. This is the standard justification in the literature.

Paired design itself protects against non-normality. de Winter (2013) showed that paired t-tests are more robust to distributional violations than independent t-tests because the pairing reduces the impact of between-subject variability. Even when raw scores are skewed, the differences between pairs are often better-behaved.

Ceiling effects attenuate variance, not introduce outliers. Your ceiling effect doesn't create extreme outliers — it compresses the upper tail. This type of non-normality (bounded, not fat-tailed) is less dangerous for the t-test than outliers would be. Your IQR check confirmed only one true outlier.

Wilcoxon only — would be non-parametric and safe, but less conventional in educational research, harder to compute effect sizes for, and less familiar to examiners. Using both is the strongest approach.

Cited in paper: @Field2018 — CLT robustness at n ≥ 30. @deWinter2013 — paired designs reduce impact of distributional violations. @LiuWang2021 — bounded score data with ceiling effects.

4 Cohen's d (Effect Size, Paired)

What question does it answer?

"Okay, the change is statistically significant — but is it big enough to matter in practice?"

Why this metric, in your specific study?

p-values don't measure magnitude. Your p = 0.017 tells you the positive direction is unlikely to be random. But with n = 31, even tiny effects can be significant if variance is low, and large effects can be non-significant if variance is high. Cohen's d decouples effect magnitude from sample size — it asks "how many standard deviations apart are the pre and post means?" regardless of n.

Your gain is small in absolute terms (+0.81/10), so you need d to contextualize. An examiner will ask "0.81 points seems tiny — does this even matter?" d = 0.399 (small-to-medium) tells them: the shift is about 0.4 standard deviations. This is non-trivial for a single-session pilot with a ceiling-compressed range. Without d, you can't answer that question.

Standardized effect size enables comparison. Future controlled studies can compare their d to your d = 0.399. If a controlled study finds d = 0.6, that's a genuine improvement. Without a standardized metric, you can't build a cumulative research program.

d = mean_diff / sd_diff is the paired version. Using the SD of the differences (not the pooled SD of pre and post separately) is correct for paired designs because the pairing removes between-subject variance that would inflate the denominator in the independent version.

Glass's Δ — uses only the control group's SD, but you have no control group. Hedges' g — corrects for small-sample bias, but your paired d already uses n−1 in the SD calculation, so the bias is negligible.

Cited in paper: @Cohen1988 — established the 0.2/0.5/0.8 benchmarks.

5 Wilcoxon Signed-Rank (Non-parametric Confirmatory)

What question does it answer?

"If I drop the normality assumption entirely and just look at whether scores tended to go up or down, does the result still hold?"

Why this test, in your specific study?

Insurance against the "your data isn't normal" attack. Your Shapiro-Wilk results (all p < 0.05) are ammunition for an examiner to question the t-test. Running Wilcoxon alongside it pre-empts that criticism: "Yes, the data is non-normal. Here's the non-parametric version. Same conclusion." This is the strongest defensive posture you can take.

Not the primary test — intentionally. If Wilcoxon were primary, you'd struggle to report an effect size (there's no standard paired effect size for Wilcoxon). By making the t-test primary (with d) and Wilcoxon confirmatory, you get both: a familiar parametric result with effect size, and a non-parametric sanity check.

Your data has 9 zero-deltas. Wilcoxon discards zeros and ranks the remaining 22 non-zero differences. With 9 ties, the t-test might be slightly affected by the clump at zero. Wilcoxon's rank-based logic handles ties naturally — it only cares about direction among those who changed.

Two tests agreeing = stronger than either alone. p = 0.017 (t-test) and p = 0.014 (Wilcoxon) converging on the same conclusion tells the examiner: "this result doesn't depend on which test I chose." This is methodologically stronger than a single test.

Sign test — even simpler (just counts positives vs negatives, ignores magnitude). Lower power, would waste information. Wilcoxon uses magnitude through ranks, so it's the better non-parametric choice here.

No specific citation needed — Wilcoxon is a standard non-parametric alternative to the paired t-test. Its inclusion is methodological best practice, not a novel contribution.

6 IQR Outlier Detection + Sensitivity Analysis

What question does it answer?

"Is my significant result being driven by one weird data point, or is it robust?"

Why this method, in your specific study?

Your delta distribution has a visible extreme case. One participant dropped from some score to some score (Δ = −6). The next largest decline is much smaller. Visually and statistically, this point stands out. You can't just ignore it — you must show the examiner you've thought about it.

IQR is rule-based, not subjective. "This looks like an outlier" is not a valid argument. IQR gives a mathematical rule: Q1 − 1.5×IQR and Q3 + 1.5×IQR. The −6 falls below the lower fence of −3.00. This is a transparent, reproducible criterion — no judgment calls.

Sensitivity, not exclusion. You keep all 31 in the main analysis. You then show the result without the outlier (n=30, t=3.520, p<0.001, d=0.643). The result not only holds — it gets stronger. This proves the finding is conservative (including the outlier makes it harder to reach significance), which makes your conclusion more trustworthy, not less.

Prevents cherry-picking accusations. If you excluded the outlier in the main analysis without showing the full-data result, an examiner could ask "did you just remove data until you got significance?" Showing both analyses transparently pre-empts this.

Z-score method (flag |z| > 3) — less robust for small samples where one outlier inflates the SD. Modified Z-score (MAD-based) — valid alternative but less common in educational research. IQR is the convention.

Result: 1 outlier at Δ = −6. Without it: t = 3.520, p < 0.001, d = 0.643. Finding is robust.

7 Spearman ρ (KB Score vs Gain)

What question does it answer?

"Do people who are better at reconstructing concept maps also show bigger reading gains?"

Why Spearman, not Pearson, for this specific pair?

Both variables are non-normal, differently. KB scores range 0%–100% with a wide spread (SD ≈ 32%, range spans the full scale). Deltas range −6 to +4 with a clump at 0 and a ceiling-compressed positive tail. Two differently non-normal variables make Pearson's linearity assumption risky — Spearman's rank-based approach is more honest.

You're testing a monotonic, not necessarily linear, hypothesis. The theory says "better structural understanding should relate to better comprehension." It does not say the relationship must be a straight line. Spearman tests for any consistent upward trend (monotonic), which is actually closer to your theoretical claim than Pearson's straight-line requirement.

Underpowered — Spearman handles this better. With n = 31, correlation tests are underpowered. Spearman is slightly more robust to the influence of extreme values in small samples because it uses ranks. A single person with KB = 100% and Δ = 0 would pull a Pearson r more than a Spearman ρ.

Pearson was used for pre-vs-gain (different question). You used Pearson for pre vs gain because that relationship is theoretically linear (higher pre → less room → lower gain). You used Spearman for KB vs gain because that relationship has no theoretical reason to be linear. Using different correlation types for different questions shows deliberate methodological thinking.

Pearson for KB vs gain — would have given a similar non-significant result but would be harder to defend given the non-normal distributions. Kendall's τ — even more robust than Spearman but less common and harder for examiners to interpret.

Result: ρ = 0.198, p = 0.284 — no significant correlation.

8 Pearson r (Pre-Score vs Gain)

What question does it answer?

"Is the ceiling effect real? Do people who started higher gain less?"

Why Pearson, not Spearman, for this pair?

The ceiling hypothesis predicts a linear negative relationship. The theory is straightforward: higher pre-score → less room to improve → smaller gain. This is naturally linear, not just monotonic. Pearson directly tests "does gain decrease proportionally as pre-score increases?" which is exactly the ceiling-effect question.

r² is meaningful here. r = −0.549 → r² = 0.30. You can tell the examiner: "About 30% of the variance in gain is explained by how high someone started." This interpretation only works with Pearson. Spearman's ρ doesn't have a clean variance-explained interpretation.

Both variables are measured on ratio scales with meaningful intervals. Pre-scores (0–10) and gains (−6 to +4) are both on the same scale with equal-interval properties. Pearson is designed for this. KB scores (%) are also ratio, but their distribution is u-shaped (clumps at extremes), which is why Spearman was used there.

Pre scores are less skewed than KB scores. Pre scores have a ceiling clump (51.6% ≥ 8) but are not as wildly distributed as KB scores (0–100%). The normality violation is milder, making Pearson less risky.

Spearman for pre vs gain — would also be significant but would lose the r² interpretation and the clean "linear ceiling effect" narrative.

Result: r = −0.549, p = 0.001 — strong negative linear relationship. Ceiling effect confirmed.

9 Median Split

What question does it answer?

"The Pearson r says high-pre people gain less — but let me show you exactly how much less, and test whether the gap is significant."

Why split at the median, in your specific study?

Pearson r alone doesn't quantify the gap. r = −0.549 tells you there's a relationship. But an examiner wants numbers: "how much did the low group gain? How much did the high group lose? Is the gap big?" The median split gives concrete group means: +1.53 vs −0.33. Much more intuitive than a correlation coefficient.

Median is a natural split point for your data. With 51.6% of participants at or above 8, and a skewed distribution, the mean (7.13) would create unbalanced groups. The median (8.0) splits the sample exactly: 19 below/at median, 12 above. Equal-sized groups maximize statistical power for the comparison.

The split itself is interpretable. Median = 8/10. In educational terms, 8 is a natural boundary — it's the difference between "needs improvement" and "already proficient." The split isn't arbitrary; it corresponds to a meaningful performance threshold.

Enables between-group inferential tests. Once you split, you can run Welch t, Mann-Whitney, and Hedges' g — a full independent-groups analysis. The correlation alone can't give you group means, p-values for the gap, or effect sizes for the difference.

Median splits lose information compared to keeping the continuous variable. An examiner may ask "why not use pre-score as a continuous predictor in a regression?" The answer: a regression would be better for prediction, but the median split is better for communication — it makes the ceiling effect visible and intuitive. Both approaches answer the same question.

10 Welch t-test + Mann-Whitney U (Group Comparison)

What question do they answer?

"Is the gap between low-pre and high-pre group gains statistically significant, or could it be random?"

Why Welch, not Student's t?

Unequal group sizes. n₁ = 19, n₂ = 12 — groups are unbalanced. Student's t assumes equal variance and becomes unreliable with unequal n. Welch's t does not assume equal variance, making it the safe default for unequal groups.

You tested the assumption anyway. Levene's test (p = 0.270) says variances are roughly equal — but you still used Welch because it's never wrong to use Welch. Using Student's t when the assumption is borderline is a methodological risk; using Welch when the assumption is met loses no power. Welch is always the conservative choice.

High group has non-normal deltas. Shapiro-Wilk for the high group: p < 0.001 — strongly non-normal. Welch's t, like the paired t, has some CLT protection, but combined with unequal n, the non-parametric backup (Mann-Whitney) becomes essential.

Student's t — requires equal variance assumption. Levene's test says it's met (p=0.270), but with n₁=19, n₂=12 and borderline normality, Welch is simply safer.

Why also Mann-Whitney U?

Same logic as paired t + Wilcoxon. Welch t = parametric, assumes approximate normality per group. Mann-Whitney = non-parametric, no distributional assumption. The high group's deltas are severely non-normal (Shapiro-Wilk p < 0.001). If the Welch t and Mann-Whitney disagree, you trust Mann-Whitney. If they agree (they do — p = 0.013 and p = 0.027), the non-normality didn't matter.

Small subgroup sample sizes. n₂ = 12 is well below the CLT threshold of 30. The high group alone cannot rely on asymptotic normality. Mann-Whitney provides a distribution-free check that doesn't need n ≥ 30.

Welch: t = 2.701, p = 0.013. Mann-Whitney: U = 168.0, p = 0.027. Both significant.

11 Hedges' g (Independent Effect Size)

What question does it answer?

"The gap between groups is significant — but how big is it, after correcting for small sample bias?"

Why Hedges' g, not Cohen's d, for the independent comparison?

Small sample bias matters at your n. Cohen's d for independent groups slightly overestimates the true population effect when samples are small. At n₁ + n₂ = 31, the bias is about 3% (correction factor = 0.974). Hedges' g applies this correction. For n > 50, d and g are virtually identical — but at 31, the distinction is methodologically visible. Reporting g instead of d shows you understand this nuance.

Note: You used Cohen's d for the paired comparison (d = 0.399) and Hedges' g for the independent comparison (g = 0.988). This is correct — paired d already uses the SD of differences with n−1, which provides its own small-sample correction. Independent d doesn't, so the Hedges correction is needed.

g = 0.988 is a key number. It tells the examiner: the gap between low-pre and high-pre groups is nearly 1 full standard deviation — a large effect by any standard. Without g, you'd just have group means and a p-value for the gap. g quantifies the magnitude of the ceiling effect.

Cohen's d (uncorrected) = 1.015 — slightly larger but overestimates. Glass's Δ — uses control SD, no control group. Cliff's delta — non-parametric effect size for Mann-Whitney, less common.

Cited in paper: @Hedges1981 — established the small-sample correction. @Cohen1988 — the benchmarks (0.2/0.5/0.8) apply to g as well as d.

12 Levene's Test (Homogeneity of Variance)

What question does it answer?

"Before I compare the two groups, are their variances similar enough, or do I definitely need Welch's correction?"

Why run it if you're using Welch anyway?

Transparency. Levene's test (p = 0.270) tells the examiner: "I checked — the variances are similar. So I could use Student's t, but I'm using Welch as a conservative choice." Without Levene's, an examiner might say "did you just assume unequal variances without checking?" Running it and reporting it pre-empts this.

Diagnostic, not decision-making. You're not using Levene's to decide between Student's and Welch — you're using Welch regardless. Levene's is reported to show that the groups are comparable in spread (low group SD = 1.78, high group SD = 1.92 — very similar). This is context for interpreting the Welch t result.

Bartlett's test — more sensitive to non-normality. Levene's is robust to moderate non-normality, which matters because the high group is non-normal.

F = 1.264, p = 0.270 — variances not significantly different.

13 TAM Construct Means vs Threshold

What question does it answer?

"Do participants lean toward agreeing that Yomilink is useful and easy to use, or are they neutral/negative?"

Why simple means, not a t-test against 3.5?

Descriptive, not inferential purpose. The TAM in this study is reported as "initial acceptance evidence" — it's descriptive. A one-sample t-test against μ = 3.5 would be possible but would overstate the claim. With n = 35, a t-test might find significance even for a practically trivial difference (e.g., mean = 3.6). The paper intentionally keeps TAM in the descriptive space to match its pilot-study framing.

3.5 is a convention, not a hard cutoff. TAM research uses 3.5 as the midpoint between Neutral (3) and Agree (4). Scores above it are interpreted as "leaning positive." This is a qualitative threshold, not a statistical one — there's no p-value for "above 3.5." Reporting the means (3.67, 3.55) and the threshold (3.5) lets the reader judge.

Per-item means are shown visually. The bar chart with the 3.5 reference line shows that all 10 items scored above the threshold. This is more informative than a single t-test — it shows which items contributed to the overall mean and whether any individual items fell below.

One-sample t-test against 3.5 — would quantify the probability that the population mean is above 3.5, but would overstate the role of TAM in a feasibility pilot. Cronbach's α for reliability — mentioned in the methodology but the actual values aren't in the defence slides.

Cited in paper: @Davis1989 — original TAM framework establishing the 5-point scale and threshold convention.

14 The Core Pattern

Question	Primary test	Why	Backup test	Why
Did scores change?	Paired t	CLT at n≥30, gives d, familiar to examiners	Wilcoxon	No normality assumption, confirms result isn't distribution-dependent
KB score vs gain?	Spearman ρ	Both vars non-normal, monotonic hypothesis	—	Single test sufficient; non-significant result anyway
Pre vs gain?	Pearson r	Theory predicts linear relationship, gives r²	—	Single test sufficient; strongly significant
Low vs high pre group?	Welch t	Unequal n, doesn't assume equal variance	Mann-Whitney	High group non-normal (n=12 < 30), distribution-free check

The pattern is: parametric primary + non-parametric confirmatory + effect size. This is the strongest defensive posture for a pilot study where data characteristics (non-normality, ceiling effects, small n for subgroups) create threats to parametric assumptions.

What to say if an examiner asks "Why didn't you just use non-parametric tests for everything?"

"Non-parametric tests are safer distributionally, but they limit what you can report. Effect sizes like Cohen's d and Hedges' g are standardized on parametric assumptions — there's no widely accepted paired effect size for Wilcoxon. Also, parametric tests are more familiar to most readers and allow direct comparison with prior KB studies, which predominantly use t-tests. By running both and showing they agree, I get the best of both worlds: interpretable effect sizes from the parametric tests, and distribution-free confirmation from the non-parametric ones."