AR-Guided Sign Language & Communication in Deaf Sports - A Narrative Review

Document Type : Review Article

Authors

1 Dayananda Sagar University, Bengaluru, India

2 Jain (Deemed-to-be University)

Abstract

Deaf athletes compete in national and international events on a regular basis, and the majority of them train in clubs alongside hearing athletes. People with deafness or substantial hearing loss claim to have a distinct cultural identity and belong to a sociolinguistic group, however the majority do not consider this a handicap. Hearing loss is linked to challenges with balance control, which can impede functioning. Athletes with hearing impairments face unique communication challenges in sports, where quick decision-making and effective coordination are crucial. AR is described as a technique that combines digital and physical information that occurs in real-time utilizing technological equipment. In particular, AR refers to the loading and merging of virtual items such as video, sound, photographs, text, 3D models, and so on into real-world perspectives. Thus, in the context of athletes who have difficulty hearing, Augmented Reality (AR) technology provides a unique solution that increases sign language interpretation, real-time game strategy communication, and referee signaling. Sign language (SL) is the primary mode of communication between hearing-impaired persons and other populations, and it is represented through both manual (body and hand gestures) and non-manual (facial expressions) characteristics. These characteristics are used to generate utterances, which transmit the meaning of words or phrases. This paper investigates the significance of AR in overcoming communication barriers for deaf athletes by combining wearable gadgets, haptic feedback, and visual overlays. By utilizing AR-driven captioning and gesture detection, deaf athletes may get rapid, context-aware information, enhancing engagement and performance.

Keywords

Main Subjects


  1. References

    1. Padden, C. A. and Humphries, T. (1988) Deaf in America. Cambridge, MA: Harvard University Press
    2. Mohammad Reza Mirzaei, Seyed Ghorshi, and Mohammad Mortazavi. 2012. Combining Augmented Reality and Speech Technologies to Help Deaf and Hard of Hearing People. In 2012 14th Symposium on Virtual and Augmented Reality. 174–181. https://doi.org/10.1109/SVR.2012.10
    3. Dhruv Jain, Bonnie Chinh, Leah Findlater, Raja Kushalnagar, and Jon Froehlich. 2018. Exploring Augmented Reality Approaches to Real-Time Captioning: A Preliminary Autoethnographic Study. In Proceedings of the 2018 ACM Conference Companion Publication on Designing Interactive Systems (Hong Kong, China) (DIS 18 Companion). Association for Computing Machinery, New York, NY, USA, 7–11.https://doi.org/10.1145/3197391.3205404
    4. Ru Guo, Yiru Yang, Johnson Kuang, Xue Bin, Dhruv Jain, Steven Goodman, Leah Findlater, and Jon Froehlich. 2020. HoloSound: Combining Speech and Sound Identification for Deaf or Hard of Hearing Users on a Head-Mounted Display. In Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, Greece) (ASSETS ’20). Association
    5. Yi-Hao Peng, Ming-Wei Hsi, Paul Taele, Ting-Yu Lin, Po-En Lai, Leon Hsu, Tzu-chuan Chen, Te-Yen Wu, Yu-An Chen, Hsien-Hui Tang, and Mike Y. Chen. 2018. SpeechBubbles: Enhancing Captioning Experiences for Deaf and Hard- of-Hearing People in Group Conversations. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/3173574.3173867
    6. Vinoba Vinayagamoorthy, Maxine Glancy, Christoph Ziegler, and Richard Schäf- fer. 2019. Personalising the TV Experience Using Augmented Reality: An Exploratory Study on Delivering Synchronised Sign Language Interpretation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300762
    7. Le Luo, Dongdong Weng, Guo Songrui, Jie Hao, and Ziqi Tu. 2022. Avatar Interpreter: Improving Classroom Experiences for Deaf and Hard-of-Hearing People Based on Augmented Reality. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI EA ’22).Association for Computing Machinery, New York, NY, USA, Article 318, 5 pages.
    8. Roshan Mathew, Garreth W Tigwell, and Roshan L Peiris. 2024. Deaf and Hard of Hearing People’s Perspectives on Augmented Reality Interfaces for Improving the Accessibility of Smart Speakers. In the International Conference on Human-Computer Interaction. Springer, 334–357. https://doi.org/10.1007/978-3-031- 60881-0_21
    9. Sanzida Mojib Luna, Jiangnan Xu, Konstantinos Papangelis, Garreth W. Tigwell, Nicolas Lalone, Michael Saker, Alan Chamberlain, Samuli Laato, John Dunham, and Yihong Wang. 2024. Communication, Collaboration, and Coordination in a Co-located Shared Augmented Reality Game: Perspectives From Deaf and Hard of Hearing People. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 36, 14 pages. https://doi.org/10.1145/ 3613904.3642953
    10. Nemček, D., & Mókušová, O. (2020). Position of sport in subjective quality of life of deaf people with different sport participation level. Physical Culture and Sport, 87(1), 1-8.
    11. Nemček¹, D., & Kručanica, L. (2014). Satisfaction with health status in people with hearing impairments. SPORTS, PHYSICAL ACTIVITY AND HEALTH, 51, 185.
    12. Uchida, W., Marsh, H. W., & Hashimoto, K. (2015). Predictors and correlates of self-esteem in deaf athletes. European Journal of Adapted Physical Activity, 21-30.
    13. Açak, M., & Kaya, O. (2016). A Review of Self-Esteem of the Hearing Impaired Football Players. Universal Journal of Educational Research, 4(3), 524-530.
    14. Brancaleone, M. P., & Shingles, R. R. (2015). Communication patterns among athletes who are deaf or hard-of-hearing and athletic trainers: a pilot study. Athletic Training & Sports Health Care, 7(1), 29-33.
    15. Rochon, W.; Feinstein, S.; Soukup, M. Effectiveness of American Sign Language in Coaching Athletes Who Are Deaf. Online Submission 2006, 1–26
    16. Acak, M. Evaluations of Deaf Team Athletes Concerning Professional Skills of Their Coaches. J. Rehabil. Health Disabil. 2015, 1, 1–11.
    17. Okan, I. Deaf Football Players’ Perception of the Coaches’ Occupational Skills. Anthropologist 2016, 25, 180–185
    18. Mesch, J., & Clark, B. (2023). Deaf sport. In Routledge Encyclopedia of Sport Studies. https://doi.org/10.4324/9780367766924-RESS70-1
    19. Osborn, M. M., & Meyer, C. M. (2023). Providing athletic training resources to Deaf and hard-of-hearing persons. Indiana State University. https://scholars.indianastate.edu/cgi/viewcontent.cgi?article=1142&context=clinat
    20. Gaweł, E., Soto-Rey, J., & Zwierzchowska, A. (2024). Trends and future directions in the sports performance of Deaf and hard-of-hearing athletes: A systematic review. Applied Sciences. https://www.mdpi.com/2076-3417/14/16/6860
    21. Jordan, J. M. and Giansant, T. (2001) ‘ Origins of CISS ’, in Lovett, J. M., Giansanti, T., and Eickman, J. (eds) CISS 2001: A Review. Redditch: Red Lizard, pp. 12–15.
    22. Clark, B. and Mesch, J. (2018) ‘ A global perspective on disparity of gender and disability for deaf female athletes ’, Sport in Society, 21(1), pp. 64–75. doi:10.1080/17430437.2016.1225808
    23. Kovoor, M., Durairaj, M., Karyakarte, M. S., Hussain, M. Z., Ashraf, M., & Maguluri, L. P. (2024). Sensor-enhanced wearables and automated analytics for injury prevention in sports. Measurement: Sensors, 32, 101054. https://doi.org/10.1016/j.measen.2024.101054
    24. Bozyer, Z. (2015). Augmented Reality in Sports: Today and Tomorrow. International Journal of Sport Culture and Science, 3(Special Issue 4), 314-325. https://doi.org/10.14486/IJSCS392
    25. Zollmann, S., Langlotz, T., Loos, M., Lo, W. H., & Baker, L. (n.d.). ARSpectator: Exploring augmented reality for sport events. University of Otago.
    26. Goebert, C. (2020). Augmented reality in sport marketing: Uses and directions. Sports Innovation Journal, 1(November), 134–151.
    27. Sano, Y., Sato, K., Shiraishi, R., & Otsuki, M. (2016). Sports support system: Augmented ball game for filling gap between player skill levels. In Proceedings of the 2016 ACM international conference on interactive surfaces and spaces (pp. 361–366).
    28.  https://virtualeye.tv/
    29. 30.   Seçkin, A.Ç.; Ate¸ s, B.; Seçkin, M. Review on Wearable Technology in Sports: Concepts, Challenges and Opportunities. Appl. Sci. 2023, 13, 10399. https://doi.org/ 10.3390/app131810399
    30. Polar UK, (2015). “Innovations | Polar UK”. Retrieved from http://www.polar.com/uk-en/ about_polar/ who_we_are/innovations.
    31. Chen, K. and Bassett, D. (2005). “The Technology of Accelerometry-Based Activity Monitors: Current and Future”, Medicine & Science in Sports & Exercise, 37 (Supplement), pp. S490-S500.
    32. Ermes, M., Parkka, J., Mantyjarvi, J. and Korhonen, I. (2008). “Detection of Daily Activities and Sports With Wearable Sensors in Controlled and Uncontrolled Conditions”, IEEE Transactions on Information Technology in Biomedicine, Vol.12, No. 1, pp. 20-26.
    33. https://scholars.indianastate.edu/cgi/viewcontent.cgi?article=1142&context=clinat
    34. Q.Liu,S.Ghodrat,G.Huisman,K.M.B.Jansen,Mater.Des.2023, 233, 112264; b) M. Zhu, S. Biswas, S. I. Dinulescu, N. Kastor, E. W. Hawkes, Y. Visell, Proc. IEEE 2022, 110, 246.
    35. Y.Huang,K.Yao,J.Li,D.Li,H.Jia,Y.Liu,C.K.Yiu,W.Park,X.Yu, Mater. Today Phys. 2022, 22, 100602;
    36. H. Bai, S. Li, R. F. Shepherd, Adv. Funct. Mater. 2021, 31, 2009364.
    37. J. Yin, R. Hinchet, H. Shea, C. Majidi, Adv. Funct. Mater. 2020, 31, 2007428.
    38. Z.Zhou,Y.Yang, J. Liu, J.Zeng,X.Wang,H. Liu, IEEE Trans. Haptics 2022, 15, 464; b) P. Kourtesis, F. Argelaguet, S. Vizcay, M. Marchal,C.Pacchierotti,IEEETrans.Haptics2022,15,479;c)K.Kim, Curr. Opin. Biomed. Eng. 2022, 21, 100368.
    39. F. Morosi,M.Rossoni,G.Caruso,Autom.Constr.2019,105,102848.
    40. Dahiya, R. S., & Valle, M. (2013). Robotic tactile sensing: technologies and system (Vol. 1). Dordrecht: Springer.
    41. Johansson, R. S., & Vallbo, Å. B. (1983). Tactile sensory coding in the glabrous skin of the human hand. Trends in neurosciences, 6, 27-32.
    42. Goethals, P., Reynaerts, D., & Van Brussel, H. (2012, May). Pneumatic tactile display controlled by a miniaturised proportional valve. In National Congress on Theoretical and Applied Mechanics, Date: 2012/05/09-2012/05/10, Location: Brussels, Belgium (pp. 1-4).
    43. Shitara, A., Namatame, M., Sarcar, S., & Ochiai, Y. (2024). HaptStarter: Designing haptic stimulus start system for deaf and hard-of-hearing sprinters. International Journal of Human-Computer Interaction. Retrieved from https://www.sciencedirect.com/science/article/pii/S1071581923001775
    44. Villamarín, D., & Menéndez, J. M. (2021). Haptic glove TV device for people with visual impairment. Sensors, 21(7), 2325. https://doi.org/10.3390/s21072325
    45. Uchida, T., Sumiyoshi, H., Miyazaki, T., Azuma, M., Umeda, S., Kato, N., ... & Yamanouchi, Y. (2019). Systems for supporting deaf people in viewing sports programs by using sign language animation synthesis. ITE Transactions on Media Technology and Applications, 7(3), 126-133.
    46. Luna, S. M., Tigwell, G., Papangelis, K., & Xu, J. (2024). Exploring visual scanning in augmented reality: Perspectives from deaf and hard of hearing users. ACM. https://doi.org/10.1145/3663548.3688535
    47. Raja S. Kushalnagar, Gary W. Behm, Aaron W. Kelstone, and Shareef Ali. 2015. Tracked Speech-To-Text Display: Enhancing Accessibility and Readability of Real-Time Speech-To-Text. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (Lisbon, Portugal) (AS-SETS ’15). Association for Computing Machinery, New York, NY, USA, 223–230.
    48. Keita Ohshiro and Mark Cartwright. 2022. How people who are deaf, Deaf, and hard of hearing use technology in creative sound activities. In Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (Athens, Greece) (ASSETS ’22). Association for Computing Machinery, New York, NY, USA, Article 66, 4 pages. https://doi.org/10.1145/3517428.3550396
    49. Kyrie Zhixuan Zhou, Weirui Peng, Yuhan Liu, and Rachel F Adler. 2024. Exploring the Diversity of Music Experiences for Deaf and Hard of Hearing People. arXiv preprint arXiv:2401.09025 (2024). https://doi.org/10.48550/arXiv.2401.09025
    50. Robin Thompson, Karen Emmorey, and Robert Kluender. 2006. The relationship between eye gaze and verb agreement in American Sign Language: An eye-tracking study. Natural Language & Linguistic Theory 24, 2 (2006), 571–604.https://doi.org/10.1007/s11049-005- 1829-y
    51. Robin L Thompson, Karen Emmorey, Robert Kluender, and Clifton Langdon. 2013. The eyes don’t point: Understanding language universals through person marking in American Signed Language. Lingua 137 (2013), 219–229. https://doi.org/10.1016/j.lingua.2013.10.002
    52. Laura J Muir and Iain EG Richardson. 2005. Perception of sign language and its application to visual communications for deaf people. Journal of Deaf studies and Deaf education 10, 4 (2005), 390–401. https://doi.org/10.1093/deafed/eni037
    53. Maria Bianca Amadeo, Andrea Excelsior, Mario Amore, Gianluca Serani, Beatriz Pereira da Silva, and Monica Gori. 2022. Face masks affect perception of happy faces in deaf people. Scientific Reports 12, 1 (2022), 12424. https://doi.org/10.1038/s41598-022- 16138-x
    54. Brittany A Blose and Lindsay S Schenkel. 2024. Facial and Body Posture Emotion Identification in Deaf and Hard-of-Hearing Young Adults. Journal of Nonverbal Behavior (2024), 1–17. https://doi.org/10.1007/s10919-024- 00458-9
    55. Luna, S. M., Xu, J., Tigwell, G., Saker, M., Chamberlain, A., Schwartz, D., & Papangelis, K. (2025). Exploring Deaf and Hard of Hearing Peoples' Perspectives on Tasks in Augmented Reality: Interacting With 3D Objects and Instructional Comprehension. ACM. https://doi.org/10.1145/3706598.3713678
    56. Bozyer, Z. (2015). Augmented reality in sports: Today and tomorrow. International Journal of Sport Culture and Science, 3(Special Issue 4), 314-325.
    57. Kajastila, Raine & Hämäläinen, Perttu. (2014). Augmented climbing: Interacting with projected graphics on a climbing wall. Conference on Human Factors in Computing Systems - Proceedings. 10.1145/2559206.2581139.

     

    1.  Hurwitz, A., & Jeffs, A. (2009). EYEPLY: Baseball proof of concept — Mobile augmentation for entertainment and shopping venues. 2009 IEEE International Symposium on Mixed and Augmented Reality - Arts, Media and Humanities, 55-56.
    2. Schweizer, H. (2014). Smart glasses: technology and applications. Student report.
    3. Poláček, R. (2020). User interface concept for smart glasses (Doctoral dissertation, Ph. D. Dissertation. University of West Bohemia. https://doi. org/10.13140/RG. 2.2. 26591.69288/1).