CSR: Small: Collaborative Research: Overheard at Home - Mitigating Overhearing of Continuous Listening Devices

Smart home devices that converse with humans are becoming popular day by day. These hands-free, Internet-connected devices continuously listen to their surroundings, so that whenever someone speaks to them, they can respond right away. In the presence of such always-listening devices at home, people are at the greatest risk of privacy invasion. For instance, a hacker can hack into these devices and listen to everything one says or every sound one makes at home. The goal of this project is to thwart such attacks and to ensure that continuously listening devices do not overhear anything that they should not.

This project takes a generalized approach to solve the overhearing problem of any acoustic sensing system. A programmable acoustic filter will be developed, which will inspect and filter out user-defined sounds before audio signals enter into an untrusted acoustic sensing system. To achieve this, the project is organized in four research tasks: (a) devising algorithms to separate a large number of sound sources on resource-constrained embedded platforms; (b) devising algorithms to count the number of active sources in real-time; (c) developing sound recognition algorithms that require very little training data; and (d) developing a custom, low-power, miniature embedded system.

Privacy-preserving mobile health applications are important to bringing personalized and low-cost healthcare to the mass of people which they can trust. The system will be deployed in two audio-based mobile health applications: asthma and sleep monitoring, where privacy is of utmost importance. Hardware and software materials developed in this project will be used in graduate and undergraduate level courses at both University of North Carolina and Columbia University. This award will support a female graduate student. Research results will be demonstrated at outreach events, as well as through online course offerings, and conference publications.

Anonymized datasets, algorithms, code, and the design of the embedded system will be shared with the research community through a website (https://overhearing.web.unc.edu/). The website is hosted at the University of North Carolina at Chapel Hill. The repository will be publicly accessible and will be kept up-to-date throughout the 3-year project. The website will remain alive beyond the project period for as long as possible.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.