Two-person final project for ADSP 32014/ON01 — Bayesian Machine Learning with Generative AI Applications (Summer 2026), by Shane Dunkle & Richard Pollitt. A public-dataset, uncertainty-first project that exercises nearly the entire syllabus. Separate from CulturalLens.

👥 Team: Shane Dunkle & Richard Pollitt · profile. Richard = data engineering / databases / scale & infrastructure / project management · Shane = Bayesian modeling + SLA interpretation. See Team & roles below.

📁 Shared project folder (Google Drive): SLAM project folder

🐙 GitHub repo: ChicagoShane/slam-final-project · live tracker (after Pages is on): https://chicagoshane.github.io/slam-final-project/

🌿 One-paragraph pitch

Using Duolingo's public SLAM dataset (millions of learner exercise responses over each learner's first 30 days), this project predicts when a language learner will make an error — and, crucially, how confident it is — so the tutor can say “I don't know, route to a human / adaptive review” instead of guessing. Three coupled Bayesian components: a hierarchical model (learners nested within native-language groups and skill/format types), Hidden Markov Models for latent acquisition stages across a learner's timeline, and a Bayesian neural network that separates aleatoric (inherent difficulty/noise) from epistemic (not-enough-data) uncertainty. The payoff is a calibrated, interpretable model of second-language learning that knows the limits of its own knowledge.

🎯 Research questions

• RQ1 (hierarchy). Does a hierarchical Bayesian model — learners nested within L1 group × skill/format — predict per-token errors with better calibration than pooled/no-pooling baselines, and what do the group-level effects reveal about L1 transfer and skill difficulty?
• RQ2 (stages). Do HMM-inferred latent acquisition stages over a learner's 30-day timeline improve error prediction and yield interpretable learning trajectories? ✅ Answered (forward CV, EN→ES): stages are interpretable (unlearned / learning / acquired) but do not reproducibly improve out-of-time prediction — reported as a negative result; the hierarchical Bayesian logistic model is retained as the final predictive model.
• RQ3 (abstention). Does decomposing aleatoric vs. epistemic uncertainty let the model abstain / route to human and improve selective accuracy (risk–coverage)?
• RQ4 (where to defer). How does epistemic uncertainty scale with data sparsity (new learners, rare skills, under-represented L1s) — i.e., where should an adaptive tutor defer?

📦 Data

SLAM 2018 (Duolingo Shared Task; Harvard Dataverse). ~2M+ tokens from ~6,000+ learners over their first 30 days, across three courses (learners of English, Spanish, French). Each token has a binary label (error / correct) plus features: user ID, L1/country, days-in-course, client (web/iOS/Android), session type (lesson/practice/test), exercise format (reverse_translate, reverse_tap, listen), response time, and per-token linguistic features (part-of-speech, morphology, dependency). The task predicts per-token error probability. Top shared-task models reached AUROC ≈ 0.86 (English) — a clear baseline to beat/match while adding calibrated uncertainty.

🧪 Methodology

Part A — Hierarchical Bayesian error model (Bayes nets + hierarchical + MCMC)

• Multilevel logistic model of P(error) with partial pooling: random effects for learner, L1 group, skill/format, and item-level linguistic features; a graphical-model (Bayes-net) view of the dependency structure.
• Fit with MCMC (NUTS in PyMC/NumPyro); scale to the full data via subsampling or ADVI/variational inference; diagnostics via ArviZ (R-hat, ESS).

Part B — Acquisition stages (Hidden Markov Models)

• Per-learner HMM over the 30-day sequence: latent states = acquisition stages, emissions = correctness/error patterns (optionally by skill). Interpret state transitions as learning progress; test whether stage membership improves prediction.
• Outcome (Wk 8, forward validation): the residual-state hybrid did not improve prediction — ungated hybrid significantly worsened NLL (+0.00055 [+0.00018, +0.00092]) and Brier in all four folds; entropy-gated hybrid was statistically indistinguishable from the logistic baseline. The earlier tiny official-dev gain was likely sampling variation / selection optimism. Decision: keep the hierarchical Bayesian logistic model for prediction; present the three-state HMM separately as a descriptive latent-learning model; skip the exact-refit hybrid confirmation.