Difficult Lessons on Social Prediction from Wisconsin Public Schools

by   Juan C. Perdomo, et al.

Early warning systems (EWS) are prediction algorithms that have recently taken a central role in efforts to improve graduation rates in public schools across the US. These systems assist in targeting interventions at individual students by predicting which students are at risk of dropping out. Despite significant investments and adoption, there remain significant gaps in our understanding of the efficacy of EWS. In this work, we draw on nearly a decade's worth of data from a system used throughout Wisconsin to provide the first large-scale evaluation of the long-term impact of EWS on graduation outcomes. We present evidence that risk assessments made by the prediction system are highly accurate, including for students from marginalized backgrounds. Despite the system's accuracy and widespread use, we find no evidence that it has led to improved graduation rates. We surface a robust statistical pattern that can explain why these seemingly contradictory insights hold. Namely, environmental features, measured at the level of schools, contain significant signal about dropout risk. Within each school, however, academic outcomes are essentially independent of individual student performance. This empirical observation indicates that assigning all students within the same school the same probability of graduation is a nearly optimal prediction. Our work provides an empirical backbone for the robust, qualitative understanding among education researchers and policy-makers that dropout is structurally determined. The primary barrier to improving outcomes lies not in identifying students at risk of dropping out within specific schools, but rather in overcoming structural differences across different school districts. Our findings indicate that we should carefully evaluate the decision to fund early warning systems without also devoting resources to interventions tackling structural barriers.


Improving Models for Student Retention and Graduation using Markov Chains

Graduation rates are a key measure of the long-term efficacy of academic...

A Predictive Model using Machine Learning Algorithm in Identifying Students Probability on Passing Semestral Course

This study aims to determine a predictive model to learn students probab...

Identifying At-Risk K-12 Students in Multimodal Online Environments: A Machine Learning Approach

With the rapid emergence of K-12 online learning platforms, a new era of...

How can Email Interventions Increase Students' Completion of Online Homework? A Case Study Using A/B Comparisons

Email communication between instructors and students is ubiquitous, and ...

Evaluating Splitting Approaches in the Context of Student Dropout Prediction

The prediction of academic dropout, with the aim of preventing it, is on...

EPARS: Early Prediction of At-risk Students with Online and Offline Learning Behaviors

Early prediction of students at risk (STAR) is an effective and signific...

A Machine Learning Approach for Detecting Students at Risk of Low Academic Achievement

We aim to predict whether a primary school student will perform in the `...

Please sign up or login with your details

Forgot password? Click here to reset