Development of a pattern recognition system for interpreting sign language using convolutional neural network

Abstract

Sign language is essential for communication among the deaf, hearing-impaired, and the general public. However, many people with normal hearing struggle to understand sign language, resulting in ineffective interactions with the deaf community in their daily activities. Recent studies have employed machine learning algorithms and computer application software solely to convert sign languages into text, aiming to address these challenges and bridge the communication gap among the deaf, hearing-impaired, and the public. This study developed a device utilizing a convolutional neural network (CNN) and an embedded system to translate sign languages into text and audio, enabling daily interactions among the deaf, hearing-impaired, and others in both private and public settings. The system comprises hardware components, including an OV2640 camera sensor to capture images of the signers, an ESP32 microcontroller to process the captured images and convert them into text and speech after being trained with a CNN model, a display screen to read the interpreted images as text, and a speaker that outputs the interpreted images as speech. To validate the proposed framework, 34,628 images were collected from a publicly available American Sign Language dataset. In this dataset, 80% of the images were used as training data, and 20% were reserved for testing. The results show a training accuracy of 99.76% but a low validation accuracy. The system's testing accuracy could be reliably improved with good lighting, a plain background, a microcontroller that has increased memory size, and a higher resolution camera, considering the achieved training accuracy.