CRII: CSR: Pervasive Gesture Recognition Using Ambient Light

As computing devices are becoming smaller, smarter, and more ubiquitous, computing has started to embed into our environment in various forms such as intelligent thermostats, smart appliances, remotely controllable household equipment, and weather based automated lawn irrigation systems. Consequently, we need new ways to seamlessly and effectively communicate and interact with such ubiquitous and always-available computing devices. A natural choice for such communication and interaction is human gestures because gestures are an integral part of the way humans communicate and interact with each other in their daily lives. This project aims at using ambient light and cheap commercial off-the-shelf light sensors to develop a gesture recognition system. The intuition behind this approach is that as a user performs a gesture in a room that is lit with light, the amount of light that he/she reflects and blocks changes, resulting in changes in the intensity of light in all parts of the room. The patterns of change in the intensity of light are different for different gestures, which can be learnt and used to recognize the gestures.

In developing the ambient light based gesture recognition system, this project has two primary objectives: (1) environment independence, i.e., making the system agnostic to the characteristics of the environment, such as different lighting conditions, and (2) user independence, i.e., making the system agnostic to the number of users in a room and their routine activities. Several challenges arise in developing the ambient light based gesture recognition system, such as automatically detecting the start and end of a gesture, removing noise from the time-series of sensor values, handling the varying time durations of the different occurrences of the same gesture, simultaneously recognizing the gestures of multiple people, and recognizing the gestures of non-stationary users. This project will not only address these and other similar challenges, but will also advance the knowledge and understanding of the use of ambient light for novel systems by yielding a theoretical foundation for modeling human gestures and routine activities using changes in the intensity of light.

The successful completion of this project will greatly benefit our society. First, this project will make the data set collected during this project publicly available for research. Second, the proposed ambient light based gesture recognition system will introduce a new and convenient way for users to interact with the computing embedded in their environments. Third, the proposed project will bridge several different communities such as systems, signal processing, machine learning, mobile computing, and human computer interaction; and foster interaction and communication among them. Fourth, the educational side of the project will integrate the research findings into the undergraduate and graduate curricula at North Carolina State University.