Hand gesture recognition based on CNN and YOLO techniques

Abstract: Communication is essential for humanity today and in the past. However, some individuals lack verbal communication due to their innate disability and physical losses from accidents. There are sign-language communication methods developed for such people to communicate. Artificial intelligence solutions are offered to remove the disadvantaged situations of people with disabilities due to communication in daily life. Nowadays, rapidly developing image processing and artificial intelligence methods are proper solutions for the problem focused on in this study. Convolution neural network techniques, which have become very popular recently, offer solutions to many problems. On the other hand, the YOLO algorithm shows very high performance in real-time object detection. In this study, we proposed a method for identifying the alphabets which each gesture delivers. This work studied hand detection on images and classification according to hand movements. The American Sign Language (ASL) standard was used as the sign language. The most recent version of YOLO, known as YOLOv5x, is used for gesture detection. Concentrating on the Static Sign-language problem, a study was conducted on the definition of hand movements. The letters “J” and “Z” are not included in the data set because movable hand signals are required. Apart from these two letters, a total number of 24 letters are classified. The proposed model achieved a training performance of 99.45% mAP@.5. Moreover, the proposed model has a performance of 97.9% mAP@.5 on the test dataset. The results demonstrate that the model's object detection performance is excellent. A statistical analysis of the training time shows that the training time has been drastically decreased, 4.5 hours with the current model as compared to the existing models in the literature.

How to cite this paper

 Helal, M., Shishah, W., Zakariah, M & Kashmeery, T. (2024). Hand gesture recognition based on CNN and YOLO techniques.Decision Science Letters , 13(4), 977-990.

Refrences

Abraham, E., Nayak, A., & Iqbal, A. (2019). Real-Time Translation of Indian Sign Language using LSTM. 2019 Global Conference for Advancement in Technology (GCAT), 1–5. https://doi.org/10.1109/GCAT47503.2019.8978343
Affonso, C., Rossi, A. L. D., Vieira, F. H. A., & de Carvalho, A. C. P. de L. F. (2017). Deep learning for biological image classification. Expert Systems with Applications, 85, 114–122. https://doi.org/10.1016/j.eswa.2017.05.039
Ameen, S., & Vadera, S. (2017). A convolutional neural network to classify American Sign Language fingerspelling from depth and colour images. Expert Systems, 34(3). https://doi.org/10.1111/exsy.12197
American Sign Language Letters Dataset. (2021). Roboflow Web Page.
ANSARI, Z. A., & HARIT, G. (2016). Nearest neighbour classification of Indian sign language gestures using kinect camera. Sadhana, 41(2), 161–182. https://doi.org/10.1007/s12046-015-0405-3
Arora, P., Chaudhary, G., Crespo, R. G., Khari, M., & Srivastava, S. (2021). Concepts and real-time applications of Deep Learning. Springer.
Cheok, M. J., Omar, Z., & Jaward, M. H. (2019). A review of hand gesture and sign language recognition techniques. International Journal of Machine Learning and Cybernetics, 10(1), 131–153. https://doi.org/10.1007/s13042-017-0705-5
Chong, T.-W., & Kim, B.-J. (2020). American Sign Language Recognition System Using Wearable Sensors with Deep Learning Approach. The Journal of the Korea Institute of Electronic Communication Sciences, 15(2), 291–298.
Daniels, S., Suciati, N., & Fathichah, C. (2021). Indonesian Sign Language Recognition using YOLO Method. IOP Conference Series: Materials Science and Engineering, 1077(1), 012029. https://doi.org/10.1088/1757-899X/1077/1/012029
Dima, T. F., & Ahmed, Md. E. (2021). Using YOLOv5 Algorithm to Detect and Recognize American Sign Language. 2021 International Conference on Information Technology (ICIT), 603–607. https://doi.org/10.1109/ICIT52682.2021.9491672
Fang, Y., Guo, X., Chen, K., Zhou, Z., & Ye, Q. (2021). Accurate and automated detection of surface knots on sawn timbers using YOLO-V5 model. BioResources, 16(3), 5390–5406. https://doi.org/10.15376/biores.16.3.5390-5406
Gupta, R., & Kumar, A. (2021). Indian sign language recognition using wearable sensors and multi-label classification. Computers & Electrical Engineering, 90, 106898. https://doi.org/10.1016/j.compeleceng.2020.106898
Jacob, M. G., Wachs, J. P., & Packer, R. A. (2013). Hand-gesture-based sterile interface for the operating room using contextual cues for the navigation of radiological images. Journal of the American Medical Informatics Association, 20(e1), e183–e186. https://doi.org/10.1136/amiajnl-2012-001212
Kakoty, N. M., & Sharma, M. D. (2018). Recognition of Sign Language Alphabets and Numbers based on Hand Kinematics using A Data Glove. Procedia Computer Science, 133, 55–62. https://doi.org/10.1016/j.procs.2018.07.008
Kang, B., Tripathi, S., & Nguyen, T. Q. (n.d.). Real-time Sign Language Fingerspelling Recognition using Convolutional Neural Networks from Depth map. ArXiv (Cornell University).
Kataria, G., Gupta, A., Kaushik, V. S., & Chaudhary, G. (2021). Emotion Recognition from Speech Signals Using Machine Learning and Deep Learning Techniques (pp. 63–73). https://doi.org/10.1007/978-3-030-76167-7_4
Kulshreshth, A., Pfeil, K., & LaViola, J. J. (2017). Enhancing the Gaming Experience Using 3D Spatial User Interface Technologies. IEEE Computer Graphics and Applications, 37(3), 16–23. https://doi.org/10.1109/MCG.2017.42
Li, G., Tang, H., Sun, Y., Kong, J., Jiang, G., Jiang, D., Tao, B., Xu, S., & Liu, H. (2019). Hand gesture recognition based on convolution neural network. Cluster Computing, 22(S2), 2719–2729. https://doi.org/10.1007/s10586-017-1435-x
Lichtenauer, J. F., Hendriks, E. A., & Reinders, M. J. (2008). Sign Language Recognition by Combining Statistical DTW and Independent Classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(11), 2040–2046. https://doi.org/10.1109/TPAMI.2008.123
Liu, H., & Wang, L. (2018). Gesture recognition for human-robot collaboration: A review. International Journal of Industrial Ergonomics, 68, 355–367. https://doi.org/10.1016/j.ergon.2017.02.004
Mitra, S., & Acharya, T. (2007). Gesture Recognition: A Survey. IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), 37(3), 311–324. https://doi.org/10.1109/TSMCC.2007.893280
Mohanty, A., Rambhatla, S. S., & Sahay, R. R. (2017). Deep Gesture: Static Hand Gesture Recognition Using CNN (pp. 449–461). https://doi.org/10.1007/978-981-10-2107-7_41
Neto, G. M. R., Junior, G. B., de Almeida, J. D. S., & de Paiva, A. C. (2018). Sign Language Recognition Based on 3D Convolutional Neural Networks (pp. 399–407). https://doi.org/10.1007/978-3-319-93000-8_45
Ng, W. L., Ng, C. K., Noordin, N. K., & Mohd. Ali, B. (2011). Gesture Based Automating Household Appliances (pp. 285–293). https://doi.org/10.1007/978-3-642-21605-3_32
Pavlovic, V. I., Sharma, R., & Huang, T. S. (1997). Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7), 677–695. https://doi.org/10.1109/34.598226
Pillai, M. S., Chaudhary, G., Khari, M., & Crespo, R. G. (2021). Real-time image enhancement for an automatic automobile accident detection through CCTV using deep learning. Soft Computing, 25(18), 11929–11940. https://doi.org/10.1007/s00500-021-05576-w
Pisharady, P. K., & Saerbeck, M. (2015). Recent methods and databases in vision-based hand gesture recognition: A review. Computer Vision and Image Understanding, 141, 152–165. https://doi.org/10.1016/j.cviu.2015.08.004
Rastgoo, R., Kiani, K., & Escalera, S. (2018). Multi-Modal Deep Hand Sign Language Recognition in Still Images Using Restricted Boltzmann Machine. Entropy, 20(11), 809. https://doi.org/10.3390/e20110809
Rautaray, S. S., & Agrawal, A. (2015a). Vision based hand gesture recognition for human computer interaction: a survey. Artificial Intelligence Review, 43(1), 1–54. https://doi.org/10.1007/s10462-012-9356-9
Redmon, J., & Farhadi, A. (2016). YOLO9000: Better, Faster, Stronger. Computer Vision and Pattern Recognition.
Roy, A. M., Bhaduri, J., Kumar, T., & Raj, K. (2023). WilDect-YOLO: An efficient and robust computer vision-based accurate object localization model for automated endangered wildlife detection. Ecological Informatics, 75, 101919. https://doi.org/10.1016/j.ecoinf.2022.101919
Roy, A. M., Bose, R., & Bhaduri, J. (2022). A fast accurate fine-grain object detection model based on YOLOv4 deep neural network. Neural Computing and Applications, 34(5), 3895–3921. https://doi.org/10.1007/s00521-021-06651-x
Sahoo, A. K., Mishra, G. S., & Ravulakollu, K. K. (2014). Sign language recognition: State of the art. ARPN Journal of Engineering and Applied Sciences, 9(2), 116-134.
Sagayam, K. M., & Hemanth, D. J. (2017a). Hand posture and gesture recognition techniques for virtual reality applications: a survey. Virtual Reality, 21(2), 91–107. https://doi.org/10.1007/s10055-016-0301-0
Sagayam, K. M., & Hemanth, D. J. (2017b). Hand posture and gesture recognition techniques for virtual reality applications: a survey. Virtual Reality, 21(2), 91–107. https://doi.org/10.1007/s10055-016-0301-0
Sharma, S., & Singh, S. (2020). Vision-based sign language recognition system: A Comprehensive Review. 2020 International Conference on Inventive Computation Technologies (ICICT), 140–144. https://doi.org/10.1109/ICICT48043.2020.9112409
Srivastava, S., Chaudhary, G., & Shukla, C. (2021). Text-Independent Speaker Recognition Using Deep Learning. EAI/Springer Innovations in Communication and Computing, 41–51.
Stergiopoulou, E., Sgouropoulos, K., Nikolaou, N., Papamarkos, N., & Mitianoudis, N. (2014). Real time hand detection in a complex background. Engineering Applications of Artificial Intelligence, 35, 54–70. https://doi.org/10.1016/j.engappai.2014.06.006
Tao, W., Leu, M. C., & Yin, Z. (2018). American Sign Language alphabet recognition using Convolutional Neural Networks with multiview augmentation and inference fusion. Engineering Applications of Artificial Intelligence, 76, 202–213. https://doi.org/10.1016/j.engappai.2018.09.006
Traore, B. B., Kamsu-Foguem, B., & Tangara, F. (2018). Deep convolution neural network for image recognition. Ecological Informatics, 48, 257–268. https://doi.org/10.1016/j.ecoinf.2018.10.002
Wikipedia Web Page, American Manuel Alphabet, https://en.wikipedia.org/wiki/American_manual_alphabet, 11/12/2021. (n.d.).
Wu, C.-H., Chen, W.-L., & Lin, C. H. (2016). Depth-based hand gesture recognition. Multimedia Tools and Applications, 75(12), 7065–7086. https://doi.org/10.1007/s11042-015-2632-3
Xiao, Q., Qin, M., & Yin, Y. (2020). Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing people. Neural Networks, 125, 41–55. https://doi.org/10.1016/j.neunet.2020.01.030
Xing, K., Ding, Z., Jiang, S., Ma, X., Yang, K., Yang, C., Li, X., & Jiang, F. (2018). Hand Gesture Recognition Based on Deep Learning Method. 2018 IEEE Third International Conference on Data Science in Cyberspace (DSC), 542–546. https://doi.org/10.1109/DSC.2018.00087
Ying Wu, & Huang, T. S. (n.d.). Human hand modeling, analysis and animation in the context of HCI. Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348), 6–10. https://doi.org/10.1109/ICIP.1999.817058
YOLOv5. (2021). GitHub Web Page - Https://Github.Com/Ultralytics/Yolov5/Wiki/Train-Custom-Data.

Journals

DSL Volumes

Keywords

Authors

Countries

Decision Science Letters

Hand gesture recognition based on CNN and YOLO techniques Pages 977-990 Download PDF

Add Reviews