As computing power and big data have revolutionised stock trading in recent years, one market player has lagged far behind: the US Securities and Exchange Commission, whose job policing the markets has been hampered by a serious technology gap.
Now the SEC is trying to catch up.
This month, the agency is in the final phases of testing software that will stream real-time trade data into its headquarters near Washington's Union Station, helping regulators better grasp the market's plumbing. The technology should go live in early 2013, at a cost of US$2.5 million (NZ$3.0m) for the year.
"What we're giving them is public data that they need in order to figure out what happened at any given moment of the trading day or reconstruct events," said Manoj Narang, 43, whose company - Tradeworx, based in Red Bank, N.J. - created the SEC's new software.
"To me, there's no substitute for the regulator having the capability to discover the answers on its own without any bias from outside parties."
But the true value of the technology will ultimately turn on whether the SEC, long known more for its lawyers than for its technologists, can lure the expertise needed to make sense of the data and use them properly.
The agency's credibility has suffered from a series of high-profile bugs and glitches in recent years that has shaken public confidence in the marketplace, such as the botched offering of Facebook's stock and the runaway trades linked to faulty computers at Knight Capital this year.
But the incident that came to define the SEC's shortcomings was the "flash crash" on May 6, 2010, when the stock market plunged nearly 1,000 points in minutes then whipsawed back up.
It took the SEC about four months to unwind the billions of orders that took place that day and issue a report of what happened. Although the SEC started collecting the data in June 2010, it could not aggregate them into a single database for analysis until three months later. Even the seemingly simple task of matching a quote with a trade price proved tough.
The wide gulf in technical prowess between the regulators and the regulated became painfully clear that year, prompting the SEC to explore hiring an outside firm that could gather up-to-the-minute market feeds from the public exchanges. It selected Tradeworx in June, but it declined to disclose how many other firms competed for the contract.
"Instead of building everything from scratch and hiring teams of developers, we wanted to buy something that's ready to go," said Gregg Berman, a senior adviser to the SEC's trading and markets division and a Princeton-trained nuclear physicist. The new system will be dubbed Midas, for Market Information Data Analytics System.
Tradeworx was no stranger to the SEC when it won the contract.
When Narang and his brother co-founded the company in 1999, during the height of the dot-com bubble, they were looking to create online personal finance tools for mom-and-pop investors. One of their first, a product aimed at helping investors calculate their margin risk, was purchased by the SEC in 2000 and displayed on its website, Narang said.
But the dot-com bubble burst, and the terrorist attacks of September 11, 2001, followed soon after, placing extreme pressure on the company. Instead of folding, Tradeworx morphed into a hedge fund in 2003 and later added a proprietary trading division, which uses high-speed, blink-of-an-eye trading techniques that rely on complex computer algorithms.
"Trading was a survival tactic for us," said Narang, who was trained in math and computer science at MIT.
The firm's next interaction with the SEC came a few months before the 2010 flash crash, when high-speed trading came under regulatory scrutiny. Concerned that the prevalence of high-frequency traders such as Tradeworx would undermine long-term investors' interests, the SEC began asking questions and inviting public input.
Thus began Narang's dialogue with the SEC, followed by testimony on Capitol Hill and plenty of media attention. The questions that regulators were posing could be answered empirically with the right technology, Narang said. The goal was to show them that they could make data-driven policy decisions.
"We weren't trying to sell anything," Narang said. In fact, Tradeworx had nothing to sell at that point. Its in-house tools were not commercialized, he said. But after the SEC started shopping around for technology in 2011, Tradeworx tweaked its software and threw its hat into the ring.
Through Tradeworx, the SEC will get up-to-the-minute feeds directly from the exchanges, where Tradeworx keeps its servers. It will also get tools needed to organize the massive data feeds, run complicated searches, spot anomalies and test market theories. It could even drill down into the orders, isolate a specific stock and track how it traded every microsecond of the day on one exchange - or all of the exchanges.
What Tradeworx will not do is analyze the data for the regulators, the SEC said.
One Tradeworx rival, a firm called Nanex, said it could have done a better job for less money. But the firm and some of its rivals were unaware that the SEC was soliciting bids, said Eric Hunsader, Nanex's chief executive. Others figure that the SEC did its due diligence, especially because it took the agency two years to select a winner.
But experts who track the agency say more needs to be done. They are eager for the launch of the "consolidated audit trail," a system that would require broker-dealers to report all their activity to a central repository and track the identities of those dealers and their clients.
The reports would capture all the orders placed on and off exchanges, including activity that takes place between broker-dealers before an order reaches an exchange. That activity is not known to the exchanges, and therefore would not be captured by the Tradeworx technology.
But with the audit trail system still years away, even the SEC's harshest critics say the Tradeworx deal is a critical first step in addressing an alarming regulatory blind spot.
"I scratch my head and say, 'How could the SEC not have had this in place already?' " said Joseph Saluzzi, co-head of a brokerage firm called Themis Trading. "Why are they still playing catch-up?"
When the SEC rolls out the Tradeworx technology, about a dozen people throughout the agency will make active use of it initially. An additional half-dozen people, including a few inside hires, will work in a new office devoted to full-time market analysis of the data. Resumes are rolling in from Wall Street, financial institutions and academic circles. And the new office could expand to more than a dozen, if the budget permits, the SEC said.
That's a big if. The agency has what former SEC chairman Mary Schapiro once described as a "volatile history of funding." While most other financial regulators operate on fees collected from the industries they oversee, the SEC gets its money from Congress, making it tough to commit to or upgrade technology.
Then there are the cultural challenges to overcome, said James Angel, a visiting professor at the University of Pennsylvania's Wharton School. The agency traditionally has ignored its economists. Its technologists ranked even lower in the pecking order, he said. The SEC has made huge strides in recent years hiring market experts. But without a clear push from the top, that culture could reassert itself.
"The SEC needs a culture that will listen to the results that all that data gives them," Angel said. "That's where the SEC has traditionally fallen on the job. . . . They have been known to come up with rules only a lawyer would love."
Narang acknowledged that the agency's in-house expertise will ultimately determine whether his technology proves useful to regulators. "You can give an electrician tools," he said. "But the tools alone don't make that person an electrician."
-The Washington Post