Extracting an individual's knowledge structure is a challenging task as it requires formalization of many concepts and their interrelationships. While there has been significant research on how to represent knowledge to support computational design tasks, there is limited understanding of the knowledge structures of human designers. This understanding is necessary for comprehension of cognitive tasks such as decision making and reasoning, and for improving educational programs.   
In this paper, we focus on quantifying theory-based causal knowledge, which is a specific type of knowledge held by human designers. We develop a probabilistic graph-based model for representing individuals' concept-specific causal knowledge for a given theory. We propose a methodology based on probabilistic directed acyclic graphs (DAGs) that uses logistic likelihood function for calculating the probability of a correct response. The approach involves a set of questions for gathering responses from 205 engineering students, and a hierarchical Bayesian approach for inferring individuals' DAGs from the observed responses. We compare the proposed model to a baseline three-parameter logistic (3PL) model from the item response theory.  
The results suggest that the graph-based logistic model can estimate individual students' knowledge graphs. Comparisons with the 3PL model indicate that knowledge assessment is more accurate when quantifying knowledge at the level of causal relations than quantifying it using a scalar ability parameter. The proposed model allows identification of parts of the curriculum that a student struggles with and parts they have already mastered which is essential for remediation.


Quantifying Individuals' Theory-Based Knowledge Using Probabilistic Causal Graphs: A Bayesian Hierarchical Approach

Division / Sponsors: 
Design Engineering Division and Computers & Information In Engineering Division


Paper Proceedings: https://asmedigitalcollection.asme.org/IDETC-CIE/IDETC-CIE2020/volume/83907

IDETC2020 - Virtual Conference

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.