Hierarchically Structured Nonintrusive Sign Language Recognition
Karl-Friedrich Kraiss
RWTH
Lehrstuhl fuer Technische Informatik

Systems for sign language recognition make often use of data gloves, which for
 most people is not an acceptable solution. On the other hand, nonintrusive
 video based systems so far can only cope with about 40 different gestures,
 which is far too little for useful applications. 

This paper presents a sophisticated picture processing approach to recognize a
 significantly larger set of gestures from a frontal monocular camera view.
 The target vocabulary comprises 152 gestures as selected from the German sign
 language.

The most difficulty part of this task is the localization of both signing hands 
in a picture, since 2-dimensional projections of the hands on the picture plane 
change continually with respect to form, position and orientation. Furthermore 
hands and face may overlap during gesturing, and gestures themselves show a great
 deal of variation.

In order to ensure correct recognition, a hierarchically structured tracking 
system for picture sequences has been devised. As a first step skin colored 
regions are identified and the largest region is assigned to the face. 
Subsequently the face is tracked using a combination of Mean-Shift Tracker and 
Active Shape Model.

Hypotheses with respect to local and temporal hand positions are formulated, 
using the face position as a reference point. Inferencing is based on a priori 
knowledge about gestures, on a biomechanical skeleton model, and on dynamic 
Kalman-Filter predictions. Overlaps among hands or among hands and face are 
resolved with an EM-Algorithm. The identified features are fed into a hidden 
markov classifier, that can handle the variations of gestures with respect to 
form and duration. The recognition accuracy achieved with this system is > 97 % 
for a vocabulary of 152 gestures, which is a great step forward if compared with 
the nonintrusive systems available so far. 

This work has been performed within the European Commission funded project 
WISDOM (Wireless information system for deaf people on the move).