Teaching assistants (TAs) are often responsible for the summative evaluation of students in introductory science, technology, engineering, and mathematics (STEM) laboratories and, as such, are at a critical nexus between students, learning, and their progression through the sciences. Troublingly, prior studies have demonstrated that TAs vary significantly in their assessment practices, often employing differing stringency criteria from their colleagues and even varying their criteria for feedback as a function of subject matter. This prior work has primarily generated detailed accounts of differences in grading practices using qualitative methods (e.g., video analysis and one-on-one interview). While invaluable, these methods are limited in their ability to provide timely analytics for course instructors to know whether they need to intervene in TA grading practices. In this study, we describe the development of a novel learning analytics dashboard to detect and visualize disparities in TA grading behavior in a large-scale online laboratory education platform (Labflow). We will present an initial analysis of grading data collected from a General Chemistry lab activity completed across multiple universities and evaluated by multiple independent TAs. We will discuss possible models for detecting grading disparity, including applications of Rasch Measurement theory to estimate rater stringency.

Automating the detection of grading disparities between teaching assistants in general chemistry laboratory activities

ACS Spring Meeting 2021

The official archive and repository for all ACS Meetings & Expositions abstracts and select posters or presentations.