The Next Breakthroughs in Augmented Reality
Apr 18 2018 | By Marilyn Harris | Photo: Timothy Lee Photographers
The Team That's Transforming AR Devices
Apr 18 2018 | By Marilyn Harris | Image: Nanfang Yu | Photos: Timothy Lee Photographers
For consumers, augmented reality (AR) holds out the promise of transforming how we live, work, and play; for the military, this technology could spell the difference between life or death. Current AR ocular devices are far from lightweight or battlefield-ready, and the race is on to develop a compact device that can present critical tactical information as a transparent overlay while freeing both a soldier’s hands and focus. Columbia Engineering will play a major role in turning this vision into a reality—recently, a dream team of experts in nanophotonics, materials science, nanofabrication, systems integration, and optics was awarded a fouryear, $4.7 million DARPA grant to invent an entirely new approach to AR glasses.
As part of its functionality, the new augmented reality glass will include the ability to project aberration-corrected RGB contextual images into the eye, as in the diagram above.
Michal Lipson, the Eugene Higgins Professor of Electrical Engineering and a pioneer in the field of silicon photonics holding more than 20 patents, is leading this interdisciplinary team to develop a novel lens material to revolutionize the way AR glasses look and function. In total, the team includes five Columbia engineers: Harish Krishnaswamy (electrical engineering), Nanfang Yu (applied physics), Alexander Gaeta (applied physics), and James Hone (mechanical engineering), in addition to Lipson. They are joined by Dmitri Basov from Columbia’s physics department, as well as researchers from UMass Amherst, Stanford University, and Trex Enterprises. The multifunctional AR glasses they’re devising, Lipson predicts, will be a “game changer.”
Current state-of-the-art AR glasses are large and cumbersome, regardless of whether they project contextual information onto the lens itself or project it onto the retina. The latter method, known as virtual retinal display (VRD), also suffers from limited frame refresh times and requires complex and heavy relay optics, including microelectromechanical mirrors, which render the glasses unwieldy and impractical in the field.
The Columbia team’s device improves on the VRD concept. Their innovative lenses will correct dynamically for an individual’s ocular aberrations, such as corneal contour, while projecting corrected contextual images—that is, the value-added AR information—directly onto the wearer’s retina.
Team members Nanfang Yu, Michal Lipson, and James Hone.
Team member Harish Krishnaswamy.
Team member Alexander Gaeta.
The innovation is in the material and the optical structures.
Even at this early stage, their experiments are yielding some intriguing results. “The innovation is in the material and the optical structures,” explains Lipson. “The aim is to be able to change the optical properties on the fly, electrically. Instead of transmitting light, the material tilts the beam, enabling high-resolution projection and the ability to detect light, with no moving parts.”
Their lens will hinge on the ultrafast generation of arbitrary waveforms, both in the visible and near-infrared (VIS and NIR) spectral ranges, which have been hampered in the past by a lack of actively tunable optical materials. The technology Lipson’s team proposes will be based on silicon nitride integrated photonics and incorporate engineered optical materials boasting extremely high electro-optic response with very low losses in the VIS/NIR ranges. In a significant advancement, a tiny laser located on the side of the glasses, perhaps in the frame, will direct the contextual information to the center of each lens. There the beam will be bent and projected onto the retina, appearing as a transparent overlay of the surroundings.
Fabrication of the material will be scalable, according to Lipson. The ultimate deliverable, in addition to dynamically adjusting for corneal aberrations, will be an ultrahigh-resolution, see-through, head-mounted display with vastly improved size, weight, power consumption, and field of view—packaged in a headset no bigger than an ordinary pair of glasses. Like many other technologies developed for government or defense applications, such practical AR glasses have potential far beyond the military. No doubt researchers are already envisioning uses for task assistance, medicine, maintenance and repair, tourism, and other scenarios yet to be dreamt of.