Understanding the capabilities and disruptive threats of big data is a must for all sophisticated finance professionals today. This article considers the development and the current state of Big Data Finance, as it specifically relates to investment advice, with an application to ETF investing.
Big Data Finance is a revolution as well as an evolution coming to prominence over several decades. Chartists and other technical analysts since at least the 1920s have utilized market data to derive upcoming patterns of prices. Perhaps the first formal breakthrough in Big Data Finance occurred in the 1980s, when companies like Bloomberg began packaging and delivering market data in large sets to many investment professionals, allowing investment advisors to quote accurate market data to their clients.
Next, the internet enabled investment advisors to receive streaming real-time or near-real-time financial data. The innovation spurred the growth of financial technologies such as electronic trading, exchange-traded funds (ETFs), as well as an explosion of exchanges and dark pools. The internet streamlined financial services' processes, and reduced financial transactions costs by a factor of 100 that were previously prohibitive. This created an influx of new clients for investment advisors.
Today, Big Data Finance is really about managing the scale of data and extracting the information within very large data sets. Today's big data is about faster, better analytics, that help extract that "needle from the haystack" using the latest data science inferences. Storing, managing and integrating ultra-large sets of streaming and historical data of all kinds is the daily work of data scientists. The scope of data being analyzed now include market data, social media data, news, regulations, announcements, and so on.
For investment advisors, understanding Big Data Finance today is particularly crucial when dealing with the following issues: ETFs, client risk preferences, news of all sorts, order execution, and sudden market effects, such as flash crashes.
For example, ETFs are a popular and low-cost investment medium, yet they may come with many risks. Issued by an investment manager that collects small fees for fund management, today's ETFs come in all shapes and sizes and seem to span most portfolios imaginable. While ETFs may seem innocuous, their proliferation facilitates stock return synchronicity—a condition whereby portfolio diversification is tossed out the window as the prices of all stocks move south at the same time.
Some researchers deduce that ETFs propagate shocks and cause instabilities in the markets, framing the discussion in the direction of the information spillover theory. Others propose alternative theories of ETF behavior, pinning responsibility for the ETF and flash crash effect on the market makers. Still other researchers link the so-called smart beta models and the ETFs among the causes of flash crashes.
