UNIVERSITY PARK, Pa. — Researchers in the College of Information Sciences and Technology recently received a three-year, $712,000 award from the National Institutes of Health to investigate a new form of prosthetic support for people with visual impairment.
The team aims to investigate whether and how computer vision can be employed in a more effective way. They plan to integrate computer vision support — a branch of artificial intelligence (AI) that trains computers to interpret and understand the visual world — with remote-sighted assistance (RSA), through which, for example, one can share their smartphone camera feed with a sighted person who can use that feed information to support interactions with the environment.
“I see our approach with respect to computer vision as human-centered AI in that we are asking how human and algorithmic capabilities can be synergistically integrated, instead of looking at human assistance and computer-based assistance as independent factors," said John Carroll, distinguished professor of information sciences and technology and principal investigator on the project.
In the project, computer vision algorithms will be designed to engage in conversations with human agents and be evaluated based on collaborative problem-solving in dynamic environments.
“We are construing vision as embedded in contexts of activity and other people,” said Carroll. “People with visual impairments are not primarily interested in recognizing objects out of context; rather, they are concerned with dynamic activity flows and interactions, just like other people.”
He added, “We want to investigate whether, when and how an integrated prosthetic approach — that is, integrating computer vision and remote sighted assistance — is effective and desirable.”
According to Carroll, the research team will work with the primary stakeholders as partners — both in their work and with the prosthetic itself. This is an action research version of the focus human-centered design usually places on human values and human experience, he said.
“We want to orient to more realistic interactions and scenarios and ask how we can enhance [users’] experiences relative to commercial RSA services they can purchase now,” said Carroll.
Carroll is working with co-principal investigator Zihan Zhou, assistant professor of IST, and senior investigator Mary Beth Rosson, professor of IST, on the study. The Happy Valley Chapter of the National Federation of the Blind and Penn State’s Accessible Technology and Information Initiative are community partners in the project.
“Optimistically, this could be a new paradigm for visual prosthetics,” concluded Carroll. “At the least, we expect it will provide new insights and perspective for research in computer vision, visual prosthetics and human-centered design.”