*How can instructors leverage assessment instruments in design, build, and test courses to simultaneously improve student outcomes and assess student learning well enough to improve courses for future students?*

*A learning statement is a structured text-based construct for students to record what they learned by reflecting on authentic immersive experiences in a semester a semester long Design-Build-Test course.  The immersive experiences include lectures, assignments, reviews, building, testing and a post-analysis of an electro-mechanical device to address a given customer need.*

*Over the past three years, in the School of Aerospace and Mechanical Engineering at the University of Oklahoma, Norman we have collected almost 30,000 learning statements from almost 400 students.  In the past few years we have analyzed this data to improve our understanding of what students have learned by reflecting on doing and thence how we might improve the delivery of the course.  *

*In an earlier paper, we described a text mining framework to facilitate the analysis of a vast number of learning statements.  Our focus, in the earlier paper was on describing the functionalities (i.e., data cleaning, data management, text analysis, and results visualization) of the framework and demonstrating one of the text quantification methods – term frequency - using the learning statements.  In this paper, we focus on demonstrating another text quantification method, namely, text similarity, to facilitate instructors gaining new insights from students’ learning statements. In text similarity we measure the cosine distance between two text vectors, and is typically used to compare the semantic similarity between documents.* *In this paper, we compare the similarity between what students learned (embodied in learning statements) and what instructors expected the students to learn (embodied in the course booklet) thus providing evidence-based guidance to instructors on how to improve the delivery of AME4163 – Principles of Engineering Design.*


Quantification of Students' Learning Through Reflection on Doing Based on Text Similarity

Division / Sponsors: 
Design Engineering Division and Computers & Information In Engineering Division


Paper Proceedings: https://asmedigitalcollection.asme.org/IDETC-CIE/IDETC-CIE2020/volume/83907

IDETC2020 - Virtual Conference

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.