Trends in Industry 4.0 show human robot collaborations (HRC) becoming ever prevalent as the enabling technology in moving manufacturing towards greater flexibility and intelligence. In HRC, humans and robots work in highly collaborative teams which possess both the humans’ capacity for highly abstract thought and flexibility and the robot’s exceptional speed and precision. A key focus in HRC is augmenting robot intelligence with the necessary “situational awareness” to safely interact with a human. &nbsp;To do so and provide viable HRC for more industries, depth cameras and computer vision (CV) techniques offer a lower cost alternative to the more expensive laser scanners or motion capture systems for workspace monitoring. However, a challenge is properly augmenting and fusing data from the sensor suite, riddled with noise, occlusion, and sparsity, to transform it into data a robot can use to make decisions and ensure safety of humans.&nbsp;The work presented in this paper develops a predictive collision detection algorithm for enhancing safety while respecting productivity in an HRC workspace.&nbsp; This algorithm serves as a real-time watchdog process during the robot execution of tasks to preemptively predict collisions with a human. The preemptive prediction can trigger reactive and proactive robot responses. This algorithm is designed to address two key challenges: 1) outputs from CV techniques are often highly noisy and incomplete due to occlusions, lighting, or any number of factors, and 2) human tracking CV approaches typically only provide a minimal set of keypoints (e.g., human joints) of the human. This noisy set of keypoints must be augmented to fully define a high-fidelity model of the human’s predicted spatial and temporal occupancy. Such information is necessary for reliable collision detection. The algorithm implements novel techniques to enhance a minimal set of data and infer a human’s swept volume in point cloud form. This form can then quickly be compared with a compatible robot model for detecting collisions in real time.&nbsp;The algorithm works as follows. First, a human skeleton tracking algorithm was implemented to provide live positional data of up to 18 points on a sensed human. This data was highly noisy and was often missing important keypoints of the skeleton under suboptimal measurement conditions. To ensure this skeleton data was reliable for use in HRC algorithms, a Kalman Filter was applied to joint angles computed from the skeleton tracking output. The Kalman Filter determined an estimation of the current skeleton joint angles and predictions of the joint angles at the next iteration, which could be used to reconstruct the skeleton model where necessary.&nbsp;&nbsp;Next, human dynamics and kinematics were leveraged to define boundary curves around the estimated human. By completing this process for every new frame of camera data, a sequence of boundary curves defining the human’s poses at multiple time steps were defined. The boundary curves between each pose were connected by surfaces, each populated with a point cloud including temporal information indicating the time at which the human occupied that position. These surfaces not only modeled the human at each calculated pose, but also interpolated the motion of the human between poses. The result was a point cloud encoding spatial and temporal information. A similar process was then completed for the robot’s predicted trajectories. The robot’s and human’s point clouds could then be quickly overlaid and investigated for spatio-temporal intersection, i.e., predicted collision. Run-time tests indicate that the algorithm could repair data to minimize skeleton model error, model the human’s swept volume, and check for collisions with the robot in approximately 30 ms. This algorithm presents an efficient and robust method by which a depth camera can provide reliable, real-time safety monitoring in an industrial workcell. The algorithm serves to enhance the robot’s situational awareness and give it the ability to forecast motion such that it can optimally respond to an unpredictable environment.

High Fidelity Human Modeling via Integrated Skeleton Tracking for Predictive Hrc Collision Detection

Paper Proceedings: https://asmedigitalcollection.asme.org/IMECE/IMECE2021/volume/85543

ASME 2021 International Mechanical Engineering Congress and Exposition (IMECE2021)

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.