Modern, large-scale manufacturing lines often rely on robots to automate repetitive, predictable, and sometimes dangerous tasks. However, many tasks involve a level of variability that robots are not equipped to handle, usually requiring human manipulation. In this paper, we present a generalized, holistic method for automated robotic arm handling of manufactured components using computer vision. In particular, we address scenarios in which a high volume of manufactured parts are moving along a conveyor belt at random locations and orientations with multiple robotic arms available for manipulation. Additionally, we present specific, tested solutions to all stages of the framework.&nbsp;We have developed a simple framework to enable efficient interaction between the robotic arms and objects moving along a conveyor system. The framework is centered around three major steps: (1) visual data capture, (2) data interpretation, and (3) command generation and output to robotic arms. Each stage is broadly defined and can be achieved using a number of tools/methods. In the visual data capture stage, a multi-component computer vision system takes in a live camera feed and exports it to an external processor. The type of camera utilized dramatically affects how data interpretation in the next stage takes place. In particular, we tested two vastly different camera systems: 3D, color, stereo-vision, Roboception cameras mounted on both mobile and stationary coordinate frames versus a stationary, low-cost Raspberry Pi 4 RGB camera.&nbsp;In the data interpretation stage, this video feed can then be interpreted using tools like 3D point clouds and object detection/tracking models to provide useful information such as object number, location, velocity, and orientation. The video feed taken from the Roboception cameras was interpreted by creating a continuously updating point cloud made up of a 2D grid overlaid with depth points at specific intervals. Meanwhile, the Raspberry Pi RGB camera feed was processed using trained machine learning models and a simple tracker. A Single Shot MultiBox Detector (SSD) model located and classified objects while the position of the objects after initial detection was followed by an OpenCV tracker to decrease processing load and increase framerate. Finally, a small convolutional neural network (CNN) was run to find parallel gripper grasping locations on the objects.&nbsp;Lastly, the command generation and output to the robotic arms stage takes the information acquired from the analysis in the data interpretation stage and turns it into instructions for robot control. This stage is highly dependent on the operating system of the robots available for the proposed setting. In our study, we utilized the KUKA KORE KR6 R700 and the KUKA LBR iiwa 14 R820. Despite being manufactured by the same company, the two types of robotic arms vary greatly in their software architecture. The KUKA KORE operates on KUKA Robot Language (KRL) which is a simplified, proprietary language that enables the user to “teach” spatial coordinates to the robot. The commands for this language are based on whether the robot has visited those locations prior which makes it less than ideal for variable situations, but effective in repeatable situations. On the other hand, the KUKA LBR iiwa 14 R820 robot uses KUKA Sunrise OS, a JAVA-based software. While less intuitive than KRL, Sunrise’s JAVA programming enables adaptation to greater variability. In order to have a unified simulation software, RoboDK was used. This is a functional robot simulator that can be used for applications such as pick and place. The software was used to model the environment with a KUKA KORE robot, a KUKA LBR iiwa robot, and a conveyor with moving parts. Due to restrictions imposed by the COVID-19 pandemic, RoboDK will enable us to validate our setup virtually prior to implementation in the lab.

Framework for Automated Robotic Arm Manipulation in Variable Industrial Environments

Paper Proceedings: https://asmedigitalcollection.asme.org/IMECE/IMECE2021/volume/85543

ASME 2021 International Mechanical Engineering Congress and Exposition (IMECE2021)

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.