IIIT-H Students Make Online Reading Easier for Hearing-Challenged People
In tests involving deaf and hard-of-hearing signers, users understood about 89 per cent of the sentences presented by the avatar.

Hyderabad:Reading online content could become easier for people with hearing disabilities, with two BTech students from the International Institute of Information Technology, Hyderabad (IIIT-H), developing a browser plugin that converts selected website text into Indian Sign Language (ISL) through a 3D animated avatar.
Developed by Masthan Sharif Shaik and Mohammed Raza, third-year students at the Centre for Visual Information Technology (CVIT), the tool allows users to select text on a webpage and click a ‘Click To Sign’ button to see the corresponding content translated into sign language. The project was supervised by professors C.V. Jawahar and Vinay Namboodiri of the University of Bath, the UK.
The tool addresses a language barrier faced by many people with hearing disabilities. “For a majority of them, sign language is their first language, while English, Hindi and Telugu are often acquired as second languages. Since Indian Sign Language has its own grammar and word order, directly reading conventional text can be difficult. Unlike systems that require expensive motion-capture equipment or powerful computers to generate signing avatars, the plugin works in a regular browser without a graphics processing unit (GPU) or a server-based AI model,” Shaik explained.
The researchers used videos from the official Indian Sign Language Research and Training Centre (ISLRTC) dictionary to reconstruct signers’ body, hand and finger movements in 3D. They also developed an annotation tool called Gizmo to correct key poses where automated reconstruction failed to capture finger movements accurately.
“Automatic reconstruction often gets fingers wrong, and in sign language a wrong finger can change the meaning. So, we built an annotation tool, Gizmo, to correct only the few key poses of each sign instead of every frame,” Raza said.
The system converts English sentence structures into ISL word order before displaying the signs. For instance, ‘He will come home’ becomes ‘He home come’. It then combines stored movements to produce a continuous signing sequence. Words without corresponding signs are spelt out letter by letter.
In tests involving deaf and hard-of-hearing signers, users understood about 89 per cent of the sentences presented by the avatar. However, the absence of facial expressions remains a limitation, as facial movements also convey grammatical information in ISL. “We will work to incorporate this in our next iteration,” Sharif said.
Prof. Jawahar said ISL continued to lack adequate resources and technology. The research group is working on both sign language synthesis, which converts text into signs, and sign language understanding, which interprets signs.
The research paper, ‘SignPlug – ISL Plugin for the Web’, has been accepted for presentation at the 17th Indian Conference on Computer Vision, Graphics and Image Processing, is scheduled to be held in Kolkata in December.

