Kinect Based face detection and expression recognition (in progress)

My most recent project is to create a robust face detection and facial expression recognition algorithm using the Kinect's depth sensor and RGB camera.

The idea behind it is that while face detection has been studied extensively in the recent years, it never reached a point of robustness suitable for everyday use. The problem is that doing face recognition using only RGB cameras the result depends greatly on environmental conditions and mainly lighting.

Kinect on the other hand offers an accurate depth sensor in addition to the RGB camera. The depth sensor works with IR light and is totally independent of lighting conditions. It may even be possible to recognize a face in total darkness!!!

Below are some screenshots of this work.

Depth Sensing:

Depth Based Segmentation

 PCA on a captured 6D vector (depth, x,y location and RGB components of a user)

Face Data Extraction:

The work is being done using OpenNI and matlab, although in the future it will be implemented entirelly in c++.