Glitches that lead to disruptions in trading such as the Flash Crash of 2010, the failed BATS IPO in March, the flubbed Facebook IPO in May and the flood of erroneous orders from Knight Capital in August are inevitable, a panel of operations and technology executives told the Securities and Exchange Commission Tuesday morning.
The challenge is for the industry to develop skills and practices that anticipate and remediate as many potential errors as possible, as the number of interrelated venues, algorithms and order types proliferate, said members of the market technology roundtable convened by the SEC in the wake of the Knight event.
“We have to accept that errors are inevitable, and instead create a prevention culture,” noted Sudhanshu Arya, managing director and global head of liquidity management technology at Investment Technology Group (ITG). “Software errors will happen, but it is more important to contain the cascading affect and the double-failures that spread the problem throughout the market.”
In fact, there may not be enough errors as it is, on which to build industry-wide prevention systems, said Dr. Nancy Leveson, Professor of Aeronautics, Astronautics and Engineering Systems at the Massachusetts Institute of Technology (MIT).
The rarity of software breakdowns actually hurts efforts at prevention because much can be learned when broken systems have to be fixed. “If that occurs just every six months or so, it’s not often enough,” she said.
The panel examining how to prevent errors through system design and operation was hosted by Robert W. Cook, Director of the SEC’s Division of Trading and Markets. The trick is to foster an environment of mutual protection that keepts one firm’s or one exchanges’ software flaw from provoking a ripple effect that could halt trading midstream or cause a market meltdown.
Such an environment could be created by making the business side of a firm or exchange aware of the complexity of their software requests and engaging in rigorous testing and risk assessment of new software, in advance of its use in live markets, the panelists said.
Jamil Nazarali, head of execution services at Citadel LLC, said it is vitally important to think of the entire chain of events that leads to a breakdown in the system, rather than examining only where the fault might lay in a particular software program. On August 1, the day of the Knight Capital problems, the first five minutes of the problem were due to the software, but the following 35 minutes were due to the software not being shut off, Nazareli said, and that is a problem of risk management and control.
“Had that been done,” he notes. “We would not be here right now.”
Indeed, this type of risk assessment should be present at the onset of the creation of any new market software, panelists said, and the management side of any firm or exchange needs to understand what it is asking of its technology developers when it envisions how it wants its software to perform.
“The business side—including the key stakeholders in the business—need to be involved in the process, said Christopher Rigg, a partner in the Global Financial Services unit of IBM.
“They need to be aware of what they’re asking and agree to the rigor and the testing that will be needed,’’ Rigg said.
IBM has been retained by Knight to review its processes and approaches to risk management. IBM also was retained by the Nasdaq Stock Market to review all its computing systems, after orders for Facebook shares were not properly handled by its IPO Cross system or in their transfer to the stock market itself.
Too often the business side of a firm or exchange may not understand the impact of even small changes in coding, said Dave Lauer, a consultant on market structure and high frequency trading from Better Markets, a nonprofit organization that promotes reform of capital markets.
“But also, this focus on simplicity can go too far,” Lauer noted. “We can simplify too much and often risk-checking can be overtaken by the desire to add speed.”
The panel discussed but did not come up with a working approach to testing new code or assessing risks to firms and markets new software trading products are launched.
Integration testing—making sure all pieces of the system work together without problems—is crucial, said Chris Isaacson, chief operating officer of BATS Global Markets. However, with 13 electronic exchanges and countless dark pools operating in the trading market, it’s become very difficult to artificially replicate such a complex and active market in a testing environment, he said.
As one solution, Isaacson and BATS advocates the use of test symbols by all exchanges in their production environments, which would allow broker-dealers to get hands-on experience with new algorithms or order types to see if any potential problems exist. Another possible solution, registering trading software, suggested by Lauer from Better Markets, could provide the SEC with an increased capacity to conduct real-time surveillance of the markets.
A “kill switch” procedure also could be used by exchanges to shut down aberrant activity. Executives of all four major national exchange operators – BATS, NYSE Euronext, Direct Edge and Nasdaq – as well as the Financial Industry Regulatory Authority said Friday they are prepared to set up such procedures, to head off a repeat of a Knight-like incident.
Citadel’s Nazareli said a kill switch would be an important tool at the exchanges, which are in the primary position to halt any problems early on.
“We don’t want a problem at a single firm to become a market problem,” he said.
Citadel has its own product, Fusebox, which sits outside its trading system and can act as a sort of kill-switch.
The Fusebox halts trading when Citadel’s systems get overloaded, similar to the way an overloaded fusebox in a home works.
