Self-training Machines

Imagine a world where all creators and makers can use computer vision to bring their ideas to life

May 03 2019 | By Allison Elliott | Vondrick photo credit: John Abbott | Collage courtesy of Carl Vondrick
Columbia Engineering For Creative Humanity stamp.


Through computer vision, machines perceive our environments, allowing consumers to ride in driverless cars or bypass checkout lines in grab-and-go stores. But current methods for creating such systems are expensive, requiring massive data sets and numerous humans to label them for use as training data. And the results can be limited when addressing the real-world complexity of everyday situations.

Carl Vondrick, assistant professor of computer science, is devising a better way to train machines—one that’s more effective and far less expensive, and thus likely to spur innovation across small businesses, startups, and the arts.

“The standard approach now creates systems that are highly specialized for specific tasks and lack versatility,” he says.

Instead, Vondrick’s method trains machines to learn by observing and interacting with the environment, using unlabeled data as inputs for pattern recognition, similar to how living organisms learn from environmental stimuli.

“Artificial intelligence has typically excelled in a closed-world system where you know everything about the environment,” says Vondrick of standard training methods. “However, we live in an open world; it’s unconstrained, vast, and diverse.”

In time, self-trained machines could even predict human behavior as they learn psychology, basic physics, and common sense—enabling more sophisticated wearables that can advise based on personal health goals, or robot assistants that are capable of helping with everyday tasks such as filling a cup of coffee.

For those with big ideas and small budgets, the significance of this research will be immense.

“Our research aims to lower the barrier of entry and make computer vision versatile so anyone can use it,” Vondrick says. “How can we make computer vision as economical as pen and paper—a tool so freely available anyone with an idea can just grab it and start creating?”

Stay up-to-date with the Columbia Engineering newsletter

* indicates required