Captioning on Glass (CoG)

Captioning on Glass is an on-going project creating an app for Google Glass with a companion Android phone app to assist the hard-of-hearing in everyday conversations.
By using both the user's Android phone and Glass as demonstrated, CoG has several advantages over using either alone.
1) The mobile phone puts a microphone next to the mouth of the speaker (hearing person), which reduces background noise.  (Glass's microphone is designed to focus on the user.)
2) By requiring the speaker to tap to speak, the quality of the transcription improves significantly.  The speaker constructs his speech better (avoiding "uhs and ums"), and the speech recognition knows when to start attending the audio.
3) The speaker can view the transcription and delete or correct it (a focus of future work for CoG).
4) Most of the network traffic uses the larger wifi and cellular antennas of the smart phone.  It also uses the smart phone's battery and processing, which helps preserve the battery life of Glass.
5) Because the hard-of-hearing user can look through Glass's head-up display, the hard-of-hearing user can focus on the speaker's lips and facial gestures as the speaker speaks and attempt to recognize the speech before the transcription is returned.  If he recognizes the speech himself, the conversation can continue right away.  If the listener misses a word or phrase, he can wait for the transcription.

Source code:

Demo video:


We are also working on another version of this app, "Translation on Glass", which will add the ability to translate between English and another language.