Modeling The World

The world is a dynamic place, and so are financial markets, and any decision-making system worth its salt needs to be able to incorporate the impact of that constant movement into its calculus as it happens. That’s the challenge Shipra Agrawal is tackling as she seeks to develop financial advising systems that learn how to make decisions keyed to the shifting state of the markets.

Agrawal, an assistant professor in the Department of Industrial Engineering and Operations Research and a member of Columbia’s Data Science Institute, has only recently applied her optimization and machine learning expertise to the field of finance; previously, her application areas of interest focused largely on Internet advertising, recommendation systems, revenue management, and resource allocation problems. “There is a rich set of challenging problems to solve in finance,” she said, with the kinetic state of the world presenting complexities similar to the dynamic nature of programmatic advertising markets.

A Markovian decision model underpins reinforcement learning, the basis for Agrawal’s portfolio management algorithm. (Image courtesy of Shipra Agrawal)

Agrawal is working on this project with Professor Xunyu Zhou and the team of doctoral and postdoc students at the Data Science Institute. To develop a decision-making model for portfolio management, she is employing reinforcement learning—an iterative, interactive trial-and-error process that necessarily includes a back-and-forth between the real and the virtual worlds. “Reinforcement learning has come a long way, with many key algorithmic developments in recent years, and the market provides the soft feedback that enables us to refine the algorithms,” she pointed out.

With constant feedback on the state of the market, the system learns patterns of how market states transition and how to react, detecting market shifts and crises and handling them successfully. The algorithm, Agrawal explained, must be designed to balance the trade-off between the learning aspect and the exploitation of that learning to achieve profits. “In finance, the decision-making space is especially complex, with regulatory and legal issues, customer attitudes, and occasional crises,” she said. The goal is a long-term strategy that leads to a profitable outcome—which would make for a particularly valuable component of a commercial asset management system.