Many industrial robots are limited to precise but highly repeatable tasks. Such robots are incapable of adapting to changes as they rely on a preset motion path provided by the user. This study focused on expanding current robotic systems’ repertoire to accomplish unpredictable tasks by utilizing a vision system. A vision system was chosen to improve robot operation due to the large amount of input data that can be received with only one camera. When paired with an industrial robot, this system will need to identify and track objects of significance to the robot’s operation. This action must be performed quickly and accurately to allow the robot to interact with its target. Due to the inherent need for versatility and rapid adaptation, researchers focused on designing a system that was cheap, robust, and could be easily upgraded to handle new objects and situations.

The vision system was built around a Raspberry Pi 4 and Pi Camera Module. A Raspberry Pi was chosen for its low price and simplicity. Object detection was performed using a TensorFlow Lite model. TensorFlow Lite was used instead of standard TensorFlow because it allowed for greater performance on a machine with low processing power. The model was run on a Coral Edge TPU to increase speed. Initially the model was trained on the Common Objects in Context (COCO) dataset. This detection model was run every \~20 frames to initialize an object tracker from the OpenCV image processing library. The tracker running on the Raspberry Pi was observed to have approximately double the processing rate of the detection model. Using the detector and tracker in conjunction led to faster and more accurate results than either system alone. Both the detection model and tracking algorithm output a rectangular bounding box that contained the object of interest. The locations of bounding box centers were tracked between frames to give each unique object an identification number that would be attributed to that object whenever it was within the camera’s view. Object velocity was also tracked by comparing object locations between frames over time.

Once object detection was achieved using the COCO dataset, researchers began training models on a local dataset to expand beyond common objects. The custom dataset consisted of playing cards as a trial. After several iterations of training, the vision system was able to detect and track various cards using the same methods applied earlier. This process can be specialized to fit any object the robot will need to interact with. The researchers have mounted the system to a KUKA Agilus robotic arm for initial testing and model verification. An object detection model will be trained specifically to this robot’s environment. Using this model, the KUKA robot will be able to make predictions regarding object location, orientation, and velocity for precise interaction. Testing will eventually lead to implementation on a full-scale industrial robot.


Object Detection and Tracking System for Industrial Robotics Using Raspberry Pi

Division / Sponsors: 
Aerospace, Bioengineering, Applied Mechanics, Design Engineering Division, Advanced Energy Systems, Manufacturing Engineering Division, Noise Control and Acoustics, Nondestructive Evaluation, Diagnosis, and Prognosis (NDPD), Nuclear Engineering Division, Process Industries Division, Fluids Engineering, Heat Transfer, Materials, Microelectromechanical Systems, Safety Engineering and Risk Analysis, Computers & Information in Engineering


Paper Proceedings: https://asmedigitalcollection.asme.org/IMECE/IMECE2020/volume/84478

IMECE2020 - Virtual Conference

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.