Engineering for a Secure Humanity: Debugging Black Box AI

Apr 20 2018 | By Allison Elliott | Photo Credit: Timothy Lee Photographers | Image Courtesy of Kexin Pei

Imagine a world where technology can help ensure public trust in innovation.

From self-driving cars to speech recognition, artificial intelligence (AI) applications make use of deep learning to replicate the decision-making process of the human brain. These advances are bringing about rapid changes in society that can pose tough questions in areas like cybersecurity, privacy, and safety. Suman Jana, assistant professor of computer science, sees these issues as catalysts for innovation.

“How do I create a tool that allows a policy maker to make good policy?” he asks.

Jana, working with colleagues at Columbia Engineering and other universities, seeks to create automated tools that search out and fix vulnerabilities in computer systems that could lead to breaches in security and privacy, while ensuring these exceptionally complex programs are safely deployed in platforms such as autonomous vehicles.

Recently, he worked with a group of researchers to create DeepXplore, a debugging tool that directly addresses the “black box” issue in deep learning, in which the intricate nature of a system often obscures how it “learned” an error.

“How do we figure out what it’s figuring out?” asks Jana. With DeepXplore, the team essentially reverse-engineered the learning process to understand the mistakes made by the AI algorithm used in self-driving cars and in other applications. They fed the system real-world inputs that were difficult or confusing in order to determine how and where the algorithm erred in the decision-making process. Their DeepXplore tool then retrained the system to fix the bug.

DeepXplore was able to detect thousands of errors in deep learning applications that had not been found before. In the future, Jana envisions agencies using this technology to measure the reliability of self-driving cars and other deep learning systems, just as the Department of Motor Vehicles currently licenses human drivers.

“This is the moment to get the right policies in place,” says Jana. “Our goal is to provide tools that regulators, developers, and manufacturers can use.”

Stay up-to-date with the Columbia Engineering newsletter

* indicates required