Enhancing YOLOv3 with RNN Models: application to American Sign Language recognition for deaf individuals
DOI:
https://doi.org/10.38152/bjtv7n4-047Abstract
As a natural language, sign language is a rich and expressive form of communication with properties similar to spoken languages. In this paper, we explore the application of transfer learning techniques to develop an intelligent system for recognizing American Sign Language. Instead of building a new deep convolutional neural network from scratch, we leveraged pre-trained models to capitalize on their advantages and reduce experimentation time. Specifically, we focused on YOLOv3 combined with advanced recurrent layers, including LSTM, GRU, and Bidirectional LSTM. The results were highly promising: YOLOv3 + Bidirectional LSTM achieved the highest accuracy, averaging over 94%, followed by YOLOv3 + LSTM at ~93%, and YOLOv3 + GRU at ~91%. These findings highlight the effectiveness of combining YOLOv3 with recurrent architectures for sign language recognition, demonstrating robust precision, recall, and F1 scores across models, with Bidirectional LSTM standing out as the top-performing configuration.
References
Vb, L., Kb, S., H, P., Abhishek, S., & T., A. (2023). An Empirical Analysis of CNN for American Sign Language Recognition. 421–428. https://doi.org/10.1109/icirca57980.2023.10220822
Fang, H. (2024). A Comparative Analysis of Convolutional Neural Networks for American Sign Language Recognition. Applied and Computational Engineering, 97(1), 133–138. https://doi.org/10.54254/2755-2721/97/20241410
Recognition of American Sign Language Using Modified Deep Residual CNN with Modified Canny Edge Segmentation. (2022). https://doi.org/10.21203/rs.3.rs-1521209/v1
Abdel Hafeez, K., Massoud, M., Menegotti, T., Tannous, J., & Wedge, S. (2024). American Sign Language Recognition Using a Multimodal Transformer Network. 89-628–X, 654–659. https://doi.org/10.1109/ccece59415.2024.10667296
Singla, N. (2023). American Sign Language Letter Recognition from Images Using CNN. International Conference on Electrical Engineering and Information Communication Technology, 1–9. https://doi.org/10.1109/ICEEICT56924.2023.10156922
Earnshaw, R. (2023). American Sign Language Letter Recognition from Images Using CNN. https://doi.org/10.1109/iceeict56924.2023.10156922
Abdullahi, S. B., Chamnongthai, K., Bolon-Canedo, V., & Cancela, B. (2024). Spatial–temporal feature-based End-to-end Fourier network for 3D sign language recognition. Expert Systems with Applications, 248, 123258.
Pathan, R. K., Biswas, M., Yasmin, S., Khandaker, M. U., Salman, M., & Youssef, A. A. F. (2023). Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network. Dental Science Reports, 13. https://doi.org/10.1038/s41598-023-43852-x
Dibba, M., & Min, C.-H. (2023). American Sign Language Recognition System Using Wearable Sensors and Machine Learning. 1–5. https://doi.org/10.1109/newcas57931.2023.10198199
Bouarara, H. A., Bentadj, C., & Rahmani, M. E. (2022). Deep Convolutional Real Time Model (DCRTM) for American Sign Language (ASL) Recognition. International Journal of Security and Privacy in Pervasive Computing (IJSPPC), 14(1), 1-13.
Thangali, A., Nash, J. P., Sclaroff, S., & Neidle, C. (2011, June). Exploiting phonological constraints for handshape inference in ASL video. In CVPR 2011 (pp. 521-528). IEEE.
More, P. B., Jadhav, A. N., Khatik, I., Singh, S., Borate, V. K., & Mali, Y. K. (2024, September). Sign language recognition using hand gestures. In 2024 3rd International Conference for Advancement in Technology (ICONAT) (pp. 1-5). IEEE.
Renjith, S., & Manazhy, R. (2024). Sign language: a systematic review on classification and recognition. Multimedia Tools and Applications, 83(31), 77077-77127.
Alaftekin, M., Pacal, I., & Cicek, K. (2024). Real-time sign language recognition based on YOLO algorithm. Neural Computing and Applications, 36(14), 7609-7624.
