Brown University develops dog-gesture-inspired system to improve robot object retrieval

PROVIDENCE, R.I. Brown University researchers have developed a system that enables robots to locate and retrieve objects by interpreting both spoken language and human pointing gestures. The approach combines a partially observable Markov decision process with vision language models to manage uncertainty when objects are hidden or environments are cluttered. It incorporates a probabilistic model of pointing gestures modeled as a cone based on eye gaze aligned with the elbow and wrist. The model was informed by research on how dogs respond to human gestures conducted in a Brown laboratory led by Associate Professor Daphna Buchsbaum. Experiments showed the system had an 89 percent success rate in identifying the correct object. A quadruped robot performed the task nearly 90 percent of the time when both language and gesture were provided, better than with either cue alone. The study will be presented March 17 at the International Conference on Human-Robot Interaction in Edinburgh, Scotland. Ivy He, a graduate student and lead author, worked with Ph.D. student Madeline Pelgrim and postdoctoral researcher Jason Liu. The project was supported by the National Science Foundation and the Office of Naval Research through the university's AI Research Institute on Interaction for AI Assistants.