Modeling and Analyzing Scorer Preferences in Short-Answer Math Questions

by   Mengxue Zhang, et al.

Automated scoring of student responses to open-ended questions, including short-answer questions, has great potential to scale to a large number of responses. Recent approaches for automated scoring rely on supervised learning, i.e., training classifiers or fine-tuning language models on a small number of responses with human-provided score labels. However, since scoring is a subjective process, these human scores are noisy and can be highly variable, depending on the scorer. In this paper, we investigate a collection of models that account for the individual preferences and tendencies of each human scorer in the automated scoring task. We apply these models to a short-answer math response dataset where each response is scored (often differently) by multiple different human scorers. We conduct quantitative experiments to show that our scorer models lead to improved automated scoring accuracy. We also conduct quantitative experiments and case studies to analyze the individual preferences and tendencies of scorers. We found that scorers can be grouped into several obvious clusters, with each cluster having distinct features, and analyzed them in detail.


page 5

page 10


Comparing Human and Automated Evaluation of Open-Ended Student Responses to Questions of Evolution

Written responses can provide a wealth of data in understanding student ...

Get It Scored Using AutoSAS – An Automated System for Scoring Short Answers

In the era of MOOCs, online exams are taken by millions of candidates, w...

Automated Scoring of Graphical Open-Ended Responses Using Artificial Neural Networks

Automated scoring of free drawings or images as responses has yet to be ...

Short Answer Grading Using One-shot Prompting and Text Similarity Scoring Model

In this study, we developed an automated short answer grading (ASAG) mod...

Balancing Cost and Quality: An Exploration of Human-in-the-loop Frameworks for Automated Short Answer Scoring

Short answer scoring (SAS) is the task of grading short text written by ...

Detecting Off-topic Responses to Visual Prompts

Automated methods for essay scoring have made great progress in recent y...

An Interpretable Deep Learning System for Automatically Scoring Request for Proposals

The Managed Care system within Medicaid (US Healthcare) uses Request For...

Please sign up or login with your details

Forgot password? Click here to reset