Dr. Matthew Beckman (Dept. of Statistics, Penn State Univ.)

04/03/20023

Abstract:  Research suggests "write-to-learn" tasks improve learning outcomes, yet constructed-response methods of formative assessment become unwieldy with large class sizes. This talk seeks to articulate the benefit of free-response tasks and timely formative assessment feedback, a roadmap for developing natural language processing (NLP) assisted feedback, and results from a pilot study establishing proof of principle. In the pilot study, several short-answer tasks completed by nearly 2000 introductory statistics students were evaluated by human raters and an NLP algorithm. Results indicate substantial inter-rater agreement using quadratic weighted kappa for rater pairs (each QWK > 0.74) and group consensus (Fleiss’ Kappa = 0.68). With compelling rater agreement, the study then introduces cluster analysis of response text as a mechanism for scalable formative assessment. The talk will conclude with implications for teaching and research building upon this work.

 

Short Bio. Matthew Beckman is an Associate Research Professor and Chair for Undergraduate Curricula in the Department of Statistics at Penn State University.  He earned a PhD in Educational Psychology with an emphasis on Statistics Education from the University of Minnesota and now leads a Statistics & Data Science Education Research Lab at Penn State and serves as Associate co-Director for Research with the Consortium for the Advancement of Undergraduate Statistics Education (CAUSE).