Does the paper cover everything the supervisor's R scripts do? Are there discrepancies?
The supervisor sent two R scripts. Here's what they do, line by line:
| Step | What it does |
|---|---|
| 1 | Read CSV, extract Pre-Test and Post-Test columns (ignoring duration) |
| 2 | Descriptive stats: n, mean(pre), sd(pre), mean(post), sd(post), mean(diff), sd(diff) |
| 3 | Shapiro-Wilk on Diff column (tests normality of differences) |
| 4 | Paired t-test (two-sided, default) — t.test(Post, Pre, paired=TRUE) |
| 5 | Wilcoxon signed-rank (two-sided, default) — wilcox.test(Post, Pre, paired=TRUE) |
| 6 | Cohen's d (paired) — cohen.d(Post, Pre, paired=TRUE) |
| 7 | Histogram of Diff + boxplot Pre vs Post |
| 8 | Outlier detection on Diff — boxplot.stats(Diff)$out |
| Step | What it does |
|---|---|
| 1 | Read CSV, compute Gain = Post − Pre |
| 2 | Descriptive stats: mean(pre), mean(post), mean(gain) |
| 3 | Pearson correlation Pre vs Gain — cor.test(Pre, Gain, method="pearson") |
| 4 | Median split: ifelse(Pre <= median, "Low", "High") |
| 5 | Group summary: n, mean(gain), sd(gain), mean(pre) per group |
| 6 | Shapiro-Wilk on Gain per group + Levene's test |
| 7 | Welch t-test (two-sided) — t.test(Gain ~ Group, var.equal=FALSE) |
| 8 | Mann-Whitney U (two-sided) — wilcox.test(Gain ~ Group) |
| 9 | Cohen's d (independent, Hedges corrected) — cohen.d(Gain ~ Group, hedges.correction=TRUE) |
| 10 | Scatter Pre vs Gain + boxplot by group |
| 11 | Outlier detection on Gain + sensitivity re-run |
| 12 | Final summary table per group |
| Supervisor's script | Paper (compute-inferential.py) | Match? |
|---|---|---|
| Descriptive stats (mean, SD of pre, post, diff) | Descriptive stats (mean, median, SD, min, max of pre, post, delta) — more complete | ✓ Covers it |
| Shapiro-Wilk on Diff | Shapiro-Wilk on Pre, Post, AND Delta — more thorough | ✓ Covers it |
| Paired t-test (two-sided) | Paired t-test (one-sided) — see discrepancy below | ⚠ Different p-value |
| Wilcoxon signed-rank (two-sided) | Wilcoxon signed-rank (two-sided) | ✓ Match |
| Cohen's d (paired) | Cohen's d (paired) | ✓ Match |
| Outlier detection on Diff | IQR outlier detection on Delta + sensitivity analysis | ✓ Match |
| Histogram + boxplot | Histogram + boxplot + dumbbell chart — more visualizations | ✓ Covers it |
| Supervisor's script | Paper (compute-inferential.py) | Match? |
|---|---|---|
| Pearson correlation Pre vs Gain | Pearson correlation Pre vs Delta | ✓ Match |
| Median split (Pre ≤ median → Low, else High) | Median split (Pre ≤ 8 → Low, Pre > 8 → High) | ✓ Match |
| Group summary (n, means, SDs) | Full group summary (mean pre, SD pre, mean post, SD post, mean delta, SD delta) | ✓ Covers it |
| Shapiro-Wilk per group | Shapiro-Wilk per group | ✓ Match |
| Levene's test | Levene's test | ✓ Match |
| Welch t-test (two-sided) | Welch t-test (two-sided) | ✓ Match |
| Mann-Whitney U (two-sided) | Mann-Whitney U (two-sided) | ✓ Match |
| Cohen's d (Hedges corrected) | Cohen's d + Hedges' g (manual correction) | ✓ Match |
| Scatter + boxplot | Scatter + boxplot in paper | ✓ Match |
| Outlier sensitivity | Outlier sensitivity (re-run t-test without outlier) | ✓ Match |
| Supervisor's R script | Your paper | |
|---|---|---|
| Paired t-test p-value | 0.034 | 0.017 |
| Reason | t.test(Post, Pre, paired=TRUE) Default is two-sided |
ttest_rel(pre, post, alternative="less") Explicitly one-sided |
| Null hypothesis | H₀: μpost = μpre (no change) | H₀: μpre ≥ μpost (scores didn't go up) |
| Alternative | H₁: μpost ≠ μpre (any change) | H₁: μpre < μpost (scores went up) |
| Still significant at 0.05? | Yes — p = 0.034 < 0.05 ✓ | Yes — p = 0.017 < 0.05 ✓ |
| Option | What changes | Risk |
|---|---|---|
| A: Make both two-sided (match supervisor) | t-test p becomes 0.034 (still < 0.05). Wilcoxon stays p = 0.014. | None. This matches the supervisor's expectation and removes the inconsistency. Easiest fix: change one line in compute-inferential.py. |
| B: Make both one-sided (with justification) | t-test p stays 0.017. Wilcoxon p becomes ~0.007. Add sentence: "Directional hypothesis justifies one-sided test." | Supervisor expects two-sided (their own R script uses defaults). They may ask why you deviated. If you can justify it with the directional RQ, it's defensible. |
| C: Keep as-is, be ready to explain | No change. Be prepared: "The t-test is one-sided because RQ1 asks whether scores increase. The Wilcoxon is two-sided as a conservative confirmatory check. Both are significant." | Examiner may notice the inconsistency. Having the explanation ready is fine — it's not fraudulent, just slightly inconsistent. |
| Analysis | In supervisor scripts? | In paper? | Why it's there |
|---|---|---|---|
| KB reconstruction score (mean, median, SD, range) | No | Yes — RQ2 | Core construct. Yomilink automatically computes this. RQ2 asks "how do learners perform on KB?" |
| Spearman ρ (KB score vs Gain) | No | Yes — RQ2 | Theoretical question: does structural understanding predict reading improvement? Tests whether the KB task and the MCQ measure the same construct. |
| Data quality / validity classification | No | Yes | 4 participants excluded (rushed). Shows the n=35 → n=31 flow. Supervisor's script reads all rows without filtering — it would include the 4 invalid. |
| Engagement analysis (pre vs post engagement levels) | No | Yes | Explains the fatigue pattern. Relevant for interpreting post-test quality and recommending session design changes. |
| TAM (PU + PEoU means vs 3.5 threshold) | No | Yes — RQ3 | Third research question: what are learner perceptions? Core for a platform-building study. |
| Qualitative feedback (thematic coding) | No | Yes — RQ3 | Triangulates TAM. "Helps learning" (24/35) and "initial confusion" (12/35) provide context for the quantitative results. |
| Num ber of propositions in goal map | No | Yes | nPropositions = 7. Needed to interpret KB score: 60.39% → ~4/7 props reconstructed. |
| Outlier-excluded sensitivity (RQ1) | Not in RQ1 script | Yes — RQ1 | Shows robustness. The −6 outlier doesn't drive the result. Supervisor's RQ2 script does sensitivity, so this extends the pattern to RQ1. |
| Detail | Supervisor | Paper | Notes |
|---|---|---|---|
| Shapiro-Wilk placement | Tests Diff only | Tests Pre, Post, AND Delta | Paper is more thorough. Testing all three is better because a paired t-test assumes normality of differences, but reporting pre and post normality gives context. |
| Descriptive stats | Mean, SD only | Mean, Median, SD, Min, Max | Paper is more complete. Median is especially important for skewed data. |
| Outlier method | boxplot.stats() (Tukey's fences) | IQR with explicit fences | Same method — both use Q1 − 1.5×IQR and Q3 + 1.5×IQR. Different implementations, identical logic. |
| Effect size naming | cohen.d(hedges.correction=TRUE) | Cohen's d then Hedges' g (manual) | Same result. R's effsize package calls it "Cohen's d with Hedges correction." Your paper separates them explicitly. Both correct. |
| Visualizations | Histogram + boxplot + scatter | Boxplot + dumbbell + histogram + scatter + bar charts | Paper has more variety. The dumbbell chart (individual changes) is especially effective for showing the ceiling pattern. |
| Median split threshold naming | Low / High | Low pre / High pre | Identical logic, different labels. |
The supervisor's RQ2 script re-runs the entire correlation and t-test without outliers as sensitivity (lines 152–159):
# Supervisor's RQ2 script df_no_out <- df %>% filter(!(Gain %in% outlier_vals)) cor.test(df_no_out$Pre, df_no_out$Gain) t.test(Gain ~ Group, data = df_no_out, var.equal = FALSE)
Your paper does outlier sensitivity for RQ1 (re-runs the paired t-test). But does it do the same for RQ2 (re-run Pearson r and Welch t without outliers)?
Your compute-inferential.py has non_outlier_mask filtering and re-runs only the paired t-test (RQ1). It doesn't re-run the Pearson correlation or the Welch t-test on the outlier-excluded data. This is a small gap. The supervisor explicitly asks you to check whether the ceiling effect finding (Pearson r, Welch t) holds without outliers.
| # | Issue | Severity | Fix |
|---|---|---|---|
| 1 | One-sided vs two-sided t-test inconsistency Paper reports p=0.017 (one-sided), supervisor expects p=0.034 (two-sided). Also inconsistent with Wilcoxon which is two-sided in the paper. |
Medium — conclusion unchanged, but supervisor will notice | Change alternative="less" → alternative="two-sided" in compute-inferential.py line 47. Re-run. Update reported p from 0.017 → 0.034. |
| 2 | RQ2 missing outlier sensitivity Supervisor's RQ2 re-runs Pearson r and Welch t without outliers. Paper doesn't. |
Low — small gap, unlikely to break anything, but supervisor expects it | Add to compute-inferential.py: after filtering outliers, re-run pearsonr and ttest_ind. Add the "no-outlier" values to the Typst output. |
To check whether your numbers match the supervisor's expected output, run their R scripts on KBresult_test.csv (if you have the file). The numbers should match within rounding — both use the same data (experiment-results.csv) and the same formulas.
If you don't have R installed, you can verify with Python:
# Verify two-sided p-value matches supervisor's expected output
python3 -c "
from scipy import stats
import csv, numpy as np
rows = list(csv.DictReader(open('paper/report/data/experiment-results.csv')))
valid = [r for r in rows if r['norm_status'].strip().startswith('Valid')]
pre = [float(r['pre_raw']) for r in valid]
post = [float(r['post_raw']) for r in valid]
t, p = stats.ttest_rel(pre, post) # two-sided
print(f't = {abs(t):.3f}, p = {p:.4f}') # should show t=2.219, p=0.0342
"