Computing and robotics advances in recent years have kindled worries that artificial intelligence (AI) someday will slip free to wreak destruction upon the world. Warnings about such scenarios can be found in recent works that don't carry a science-fiction label, such as documentary filmmaker James Barrat's recent book Our Final Invention: Artificial Intelligence and the End of the Human Era.
No less a scientific name than Stephen Hawking has been raising alarms about out-of-control AI. In a recent opinion piece, Hawking and several scientist co-authors warned: "One can imagine such technology outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand. Whereas the short-term impact of AI depends on who controls it, the long-term impact depends on whether it can be controlled at all."
In an era of grandmaster-beating chess programs and driverless-car prototypes, it is no surprise that anxieties about where such technology is heading have gained traction. Nor is it a surprise that critics target the tech sector and the Pentagon (especially the Defense Advanced Research Projects Agency, DARPA) for working on increasingly powerful systems that may one day outsmart, outrank or outlive us all. (A UN meeting in Geneva in May convened experts to discuss emerging "lethal autonomous weapons.")
What might be surprising is the idea that Wall Street could give rise to dangerous AI—not in the obvious sense that financial institutions raise capital for the tech sector, but in that financial technology itself might be the matrix for the rise of the machines. That is the gist of one emerging line of thinking about the dangers of smart computers.
Funding Consciousness
In Our Final Invention, Barrat presents a dark vision of AI outstripping and endangering humanity—as soon as the next few decades. DARPA and Google figure prominently in this jeremiad, but Barrat also includes a scenario sketched out by Alexander Wissner-Gross, a scientist-engineer with affiliations at Harvard and MIT, in which a powerful AI could emerge from agent-based financial models, which simulate the behavior of multiple players in a market or economy.
After noting that a huge amount of money and brainpower now goes into developing ever-better models used by hedge funds and in high-frequency trading, Barrat writes: "Wouldn't the next logical step be to make your hedge fund reflective? That is, perhaps your algorithm shouldn't automatically trigger sell orders based on another fund's massive sell-off (which is what happened in the flash crash of May 2010)."
He continues: "Instead, it would perceive the sell-off and see how it was impacting other funds, and the market as a whole, before making its move. It might make a different, better move. Or maybe it could do one better, and simultaneously run a very large number of hypothetical markets, and be prepared to execute one of many strategies in response to the right conditions."
Barrat explains: "In other words, there are huge financial incentives for your algorithm to be self-aware—to know exactly what it is and model the world around it."
How do you program self-awareness into a computer? Nobody knows, and that may be a big obstacle to this scenario coming true. Wissner-Gross suggests it might happen by accident, as the interplay of multiple algorithms gives rise to an "artificial general intelligence" (AGI), a system with a human-level grasp of the world. He tells Barrat: "If you follow the money, finance has a decent shot at being the primordial ooze out of which AGI emerges."
