SignIT!
Company
Microsoft Research, Bengaluru
Duration
1 Year, 2020 - Ongoing
Team
Roshni Poddar, Pradyumna Y M, Divya Prabha, Tarini Naik, Pratyush Kumar, Mohit Jain, Manohar Swaminathan, Punyat Tripathi, Nabeel TP, Hemanth Reddy Yeddula
My Role
Research, UX Design, Prototyping, Application Design
Tools
Adobe XD, Illustrator, Photoshop, InDesign
Publication
SignIt! An Android Game for Sign Bilingual Play [Poster]
The 25th International ACM SIGACCESS Conference on Computers and Accessibility [ASSETS’23]
SignIt! An Android Game for Sign Bilingual Play that Collects Labelled Sign Language Data [Conference]
ITU Journal on Future and Evolving Technologies [ITU’23]
The Challenges
The global Deaf and Hard-of-Hearing (DHH) community, with over 430 million individuals, relies heavily on sign language, yet faces significant challenges, especially in India, where 63 million people are DHH. Factors such as delayed diagnosis, stigma, and linguistic diversity hinder sign language and English learning.
Digital games are emerging as an effective tool for spoken and sign language education, offering enhanced engagement and socialization. However, the development of advanced machine learning models for sign language recognition is hampered by the limited availability of labeled data, which remains a critical barrier to improving accessibility for signers.
The Approach
The project's approach, known as Ludic Design for Accessibility, keeps play and playfulness central to every accessibility solution. The success of this design philosophy is measured by parameters that are amenable to measurement, which collectively form a discrete set of objectives. This approach effectively decouples these objectives from the holistic complexities of living.
Our Partner
The National Institute of Speech and Hearing (NISH), located in Thiruvananthapuram, Kerala, is an autonomous organization dedicated to the education and rehabilitation of individuals with speech and hearing impairments. NISH is actively involved in research and advocacy for the rights and inclusion of people with communication disorders
How might we leverage sign language and English learning for the DHH community in India, addressing stigma, language diversity, and the scarcity of labelled sign language data through games?
Reasearch
I learned Indian Sign Language (ISL) from Sandeep and Arun, two deaf teachers from National Institute of Speech and Hearing. Our approach integrated playful learning and teaching ISL to colleagues, fostering a more inclusive environment at MSR. Interacting with two deaf teachers, and three deaf interns deepened my understanding of Deaf culture, community experiences, Deaf Gains and the critical need for accessibility and support for the challenges faced by the deaf community.
MSR employees learn Indian Sign Language in a fun and engaging way through storytelling.
Multiple digital sign language learning games were explored with three Deaf interns, focusing on smartphone games that support inclusive play for both hearing and DHH players. The goal was to ensure that language proficiency advantages alternated between hearing and DHH individuals.
Below are a few games explored by our team consisting of three Deaf interns and 3-5 hearing team members.
Storytelling with Comic Strips
Storytelling is a powerful tool for language learning, offering vocabulary and world knowledge. We used English comic books to teach both ISL and English. During Zoom sessions, a Deaf signer narrated comic strips, helping hearing and Deaf participants improve their ISL and English skills. These sessions were recorded for revision and to extract labeled continuous sign language data.
ISL Taboo
deaf interns shared their experience playing an accessible version of Taboo during college. This game was recreated over Zoom, where one player describes a word or image without using specific signs, monitored by a competing team member. While fun, the game required basic ISL competency, limiting novice ISL hearing participants' involvement.
Memory Game
Each player adds a new object to an imaginary bag, recalls the sequence of objects already inside, and then passes the bag to the next player. Novice ISL users often used fingerspelling, with repetition helping them memorize everyday signs. The recorded sessions could provide valuable isolated sign language data.
WhatsApp Quiz
We used asynchronous WhatsApp quizzes where players signed and shared ISL-based multiple-choice questions. Participants responded with signed questions, choices, and their answers in ISL, plus the English text translations of the signed videos. The quiz creator then revealed answers and updated the scoreboard. This method enhanced ISL and English learning and produced useful labeled ISL videos, but managing the process was tedious.
This experience also led to the discovery of Kahoot! by some interns, highlighting varying levels of familiarity with gaming platforms among team members.
Key Insights
1. Developing a sign language equivalent version of Kahoot! can effectively teach ISL and English to both Deaf and hearing individuals in an enjoyable way.
2. Group play is more engaging than individual play as it encourages competition and collaboration among participants.
3. Indian Deaf and Hard of Hearing community exhibits significant diversity in language proficiency, influenced by factors such as educational background, age of exposure to sign language, and the prevalence of local languages in different regions of India.
4. The scarcity of labelled sign language data hinders technological and educational advancements for the Deaf and Hard of Hearing (DHH) community, primarily due to complexities in sign language diversity, limited funding, ethical considerations, collaboration challenges, and data quality standards.
Prototype
Prototyped SignIt through iterative design, significantly refining accessibility and usability for the Indian Deaf and Hard of Hearing (DHH) community.
Initial Attempt: A Sign Language Quiz prototype was created with questions and four options, including English text translations for each sign language video. However, during testing, players were not interested in watching the video due to the visible English text.
Revised Approach: The Sign Language Quiz prototype was redesigned with questions and four options, providing English text translations as optional hints. Participants watched the video first and could use the hint only if they needed help understanding the sign language content.
SignIt!
SignIt!, an accessible and inclusive quiz game platform that facilitates the learning of English and Indian Sign Language (ISL) with a secondary goal of data collection.
Key Features
When a player starts a quiz, they are directed to the question screen where they see a question and answer options as sign language videos. Each video has a hint button to toggle captions, which help players learn new vocabulary. Each option has a radio button for selection of the answer. After an answer is chosen, the analysis screen displays the question with the correct option highlighted in green and the incorrect option in red.
1. Individual mode
In individual mode, players play quizzes alone and can replay them multiple times. We decided not to set a time limit on the questions, as research on Kahoot! found that time limits reduced player reflection and led to rushed guesses.
2. Group mode
The purpose of groups is to collate multiple people and quizzes to introduce a social element of competition among players. Similar to individual mode, group quizzes do not have a timer and may be played asynchronously. The cumulative points earned from group quizzes are showcased on the group leaderboard.
3. Live mode
Live mode allows multiple players to participate in synchronous quizzes, where all players answer each question simultaneously.
4. Create Quiz
Players can create their own quizzes by adding the quiz name, tags, sign language, caption language, and a list of questions. In addition, they can add a quiz image, description, and set its visibility (public or private).
User Study
- 22 participants engaged with the SignIt! App, answering 1,769 solo questions and 391 in groups, and creating 82 quizzes with 210 questions.
- Feedback was positive, highlighting the app’s intuitive design and effectiveness in language learning and social interaction. Notably, four users learned sign language through the app.
- Participants took part in 132 group quizzes and 56 live quizzes, forming 47 groups and sharing 77 quizzes.
- The app gathered 2,936 labeled videos, verified by ISL experts, showcasing its potential for large-scale data collection.
The Impact
- The app enhances learning across subject matter, English vocabulary, and ISL signs, fostering social collaboration and mixed-ability interactions.
- It offers opportunities for improved dataset generation for sign-language recognition, advancing machine learning models.
- Users can create valuable sign language videos, contributing to more accurate recognition tools.
Reflection
Cultural Sensitivity in Sign Language: SignIT! exposed the complexities of sign language as more than linguistic translation; it’s deeply tied to cultural identity. I learned the importance of designing tools that respect regional and cultural diversity in sign languages, rather than imposing a standardized form. This approach ensures the tool reflects the rich cultural heritage of its users while promoting inclusivity.
Critical Engagement with ‘User Needs’: The project prompted a re-evaluation of ‘user needs,’ traditionally defined through an able-bodied lens. Instead of assuming deaf individuals need to assimilate into a hearing world, I reflected on whether design should challenge societal norms. This led to a focus on designing for transformation—creating spaces that celebrate diverse communication, not just facilitating adaptation.
Challenging the Technocentric Approach to Accessibility: The SignIT! project revealed that technology alone can't address systemic issues like the shortage of sign language interpreters or social stigmas. This shifted my understanding of accessibility, emphasizing the need for multi-layered solutions that integrate educational resources, and community involvement, beyond what tech can solve.
Interrogating the Epistemology of Design: Whose Knowledge Counts?: Engaging with the deaf community through SignIT! challenged my assumptions about whose knowledge matters in design. I realized that traditional design processes often exclude marginalized perspectives. This deepened my approach, recognizing lived experiences as essential expertise, and shifting toward co-creating solutions that reflect the realities of those directly impacted.
