Smartphones and tablets are become the worldwide tools for easily communicating among people. These devices allow textual and visual communications by means of touchscreens with which people can interact using fingertips. This interaction style requires the ability to accurately touch small areas of the screen for correctly executing a task. Accurate touches with fingers can be very difficult for people who have hand motor skills limits. Limited hand movements may be caused by arthrosis in elderly people or more serious health complications, such as a brain stroke. In these cases, the traditional use of a touchscreen to type text is very difficult and, for the worst cases, impossible. The introduction of an interaction style based on simpler gestures should help elderly people with arthrosis and patients with a brain stroke. To achieve this aim, innovative human body pose estimation techniques can be exploited to track movements of the whole human body or parts of it. This research work aims to develop an application that generates text by drawing alphanumeric characters with air gestures performed using an arm instead of a single finger. The application embeds a combination of well-known computer vision and convolutional neural networks software modules to detect and drawn alphanumeric characters and recognizing them. The whole software development is based on free software development tools and open source software development kits.

The application detects the wrist as pen tip to draw characters and automatically recognize them to sentences. Exploiting the power of the new pose-detection algorithms (e.g., OpenPose) it is easy to identify in a 2D image the key points of a human body. With the word key points, we identify the main landmark of a human body like the shoulders, the wrists, the elbows, the hips, the knees and the ankles. For the specific application, the movements made by the wrist (left or right, depending on the patient) is captured and the trajectory is used as input of another Convolutional Neural Network. The purpose of the second stage is to classify the hand-written letter and translate it in text.  The sentences are then created in the application. The spacing between one character and the following one is made simply by stopping the hand in the same position for a fixed amount of time.

First, the paper describes the scientific background relative to the use of air gesture to write text and the exploited technologies. Then, the software architecture is described by highlighting the free and open-source tools used to develop the mobile application. The tests are described and the reached results discussed. Conclusion and future works are presented.


Text Writing Using Air Gestures for People With Limited Hand Motor Skills

Division / Sponsors: 
Aerospace, Bioengineering, Applied Mechanics, Design Engineering Division, Advanced Energy Systems, Manufacturing Engineering Division, Noise Control and Acoustics, Nondestructive Evaluation, Diagnosis, and Prognosis (NDPD), Nuclear Engineering Division, Process Industries Division, Fluids Engineering, Heat Transfer, Materials, Microelectromechanical Systems, Safety Engineering and Risk Analysis, Computers & Information in Engineering


Paper Proceedings: https://asmedigitalcollection.asme.org/IMECE/IMECE2020/volume/84478

IMECE2020 - Virtual Conference

Supporting the need among researchers worldwide for additional online and multimedia content, ASME now offers the ASME Conference Video Collection. This platform will collect and provide long-term access to conference presentation videos, encompassing all technology areas within ASME's technical conference and events program. About 30,000-40,000 engineers across all mechanical engineering and related disciplines participate in ASME conferences each year. This includes authors of approximately 6,000 to 8,000 technical papers published by ASME, the majority of whom also submit video presentations for our virtual conferences. The video collection will be available as an add-on to existing product packages or as a separate subscription.