Joining Hands

The Covid-19 pandemic quickly rendered many workers denizens of the web, interacting through email and video chat. But not everyone.

“White-collar workers could stay distanced and meet through Zoom or Skype,” says Shuran Song, assistant professor of computer science, but a lot of factory and warehouse workers “don’t have the right tools to perform physical tasks remotely.” Song leads a project, funded by a five-year, $3.7 million grant from the National Science Foundation, that could transform physical labor going forward. It aims to let people with little robotics expertise direct robots from anywhere.

The plan is for workers to wear virtual reality (VR) headsets and hold VR controllers, similar to those used for VR games, as they gesture and tell a remote robot what to do. This vision lies between two extremes. At one end, automated assembly lines have been manufacturing cars and other single-issue products at high volume for decades. But robots developed for those highly regimented tasks lack the flexibility needed to handle different goods at the same time. At the other end, researchers have been developing telerobotics, in which an operator remotely controls a machine’s every movement. But some tasks require inhuman precision and consistency.

“Our project sits between the ends of the spectrum,” says Steve Feiner, professor of computer science and one of Song’s collaborators. “It augments users’ skills without requiring them to have any specialized understanding of the system. You merely motion that tab A goes into slot B. The system figures out what you want, then determines how to accomplish it.”

Song, an expert in computer vision, will work with her students to help the system understand its environment using cameras. Feiner’s team will develop its user interface, and the lab of Matei Ciocarlie, associate professor of mechanical engineering, will perfect its sense of touch.

They’re developing multi-fingered hands with tactile sensors so that the robot can “poke and prod things,” Feiner says, to see how rigid or heavy they are.

To pull all that off, Ciocarlie notes, the system will require at least two kinds of intelligence: abstract and embodied. The former tells the robot that it needs to move part C into place before joining parts A and B. The latter guides physical interaction in a rough-and-tumble world. “To be robust and fast is very hard,” Ciocarlie says, “and it’s intelligence of the first rank.”

Of course, the intelligence will be artificial. In addition to computer vision, the system will use a form of machine learning called reinforcement learning, in which a program learns to reach a goal through trial-and-error. It will also use something called inverse reinforcement learning, in which the system infers a goal from observing behavior, in this case the operator’s.

Besides shielding factory and warehouse workers from exposure to a potentially deadly virus, remote robotic collaboration offers other advantages: It can improve safety, eliminate commutes, allow companies to hire people anywhere, and level the playing field for people with disabilities.

The system will become more automated over time as it learns, and it may herald a new era for building and moving things. “If we can harness this, then the robot and the artificial intelligence are tools at the human’s disposal,” Ciocarlie says. “We can bring back into the workforce people who otherwise wouldn’t have been able to participate.”