Today read about some of the highlights and news that came from Tuesday’s Scurities and Exchange Commission Market Technology Roundtable. Traders Magazine presents three stories that cover the event from start to finish.
SEC Told Errors Inevitable. ‘Prevention Culture’ Needed.
Glitches that lead to disruptions in trading such as the Flash Crash of 2010, the failed BATS IPO in March, the flubbed Facebook IPO in May and the flood of erroneous orders from Knight Capital in August are inevitable, a panel of operations and technology executives told the Securities and Exchange Commission Tuesday morning.
The challenge is for the industry to develop skills and practices that anticipate and remediate as many potential errors as possible, as the number of interrelated venues, algorithms and order types proliferate, said members of the market technology roundtable convened by the SEC in the wake of the Knight event.
“We have to accept that errors are inevitable, and instead create a prevention culture,” noted Sudhanshu Arya, managing director and global head of liquidity management technology at Investment Technology Group (ITG). “Software errors will happen, but it is more important to contain the cascading affect and the double-failures that spread the problem throughout the market.”
In fact, there may not be enough errors as it is, on which to build industry-wide prevention systems, said Dr. Nancy Leveson, Professor of Aeronautics, Astronautics and Engineering Systems at the Massachusetts Institute of Technology (MIT).
The rarity of software breakdowns actually hurts efforts at prevention because much can be learned when broken systems have to be fixed. “If that occurs just every six months or so, it’s not often enough,” she said.
The panel examining how to prevent errors through system design and operation was hosted by Robert W. Cook, Director of the SEC’s Division of Trading and Markets. The trick is to foster an environment of mutual protection that keepts one firm’s or one exchanges’ software flaw from provoking a ripple effect that could halt trading midstream or cause a market meltdown.
Such an environment could be created by making the business side of a firm or exchange aware of the complexity of their software requests and engaging in rigorous testing and risk assessment of new software, in advance of its use in live markets, the panelists said.
Jamil Nazarali, head of execution services at Citadel LLC, said it is vitally important to think of the entire chain of events that leads to a breakdown in the system, rather than examining only where the fault might lay in a particular software program. On August 1, the day of the Knight Capital problems, the first five minutes of the problem were due to the software, but the following 35 minutes were due to the software not being shut off, Nazareli said, and that is a problem of risk management and control.
“Had that been done,” he notes. “We would not be here right now.”
Indeed, this type of risk assessment should be present at the onset of the creation of any new market software, panelists said, and the management side of any firm or exchange needs to understand what it is asking of its technology developers when it envisions how it wants its software to perform.
“The business side—including the key stakeholders in the business—need to be involved in the process, said Christopher Rigg, a partner in the Global Financial Services unit of IBM.
“They need to be aware of what they’re asking and agree to the rigor and the testing that will be needed,’’ Rigg said.
IBM has been retained by Knight to review its processes and approaches to risk management. IBM also was retained by the Nasdaq Stock Market to review all its computing systems, after orders for Facebook shares were not properly handled by its IPO Cross system or in their transfer to the stock market itself.
Too often the business side of a firm or exchange may not understand the impact of even small changes in coding, said Dave Lauer, a consultant on market structure and high frequency trading from Better Markets, a nonprofit organization that promotes reform of capital markets.
“But also, this focus on simplicity can go too far,” Lauer noted. “We can simplify too much and often risk-checking can be overtaken by the desire to add speed.”
The panel discussed but did not come up with a working approach to testing new code or assessing risks to firms and markets new software trading products are launched.
Integration testing—making sure all pieces of the system work together without problems—is crucial, said Chris Isaacson, chief operating officer of BATS Global Markets. However, with 13 electronic exchanges and countless dark pools operating in the trading market, it’s become very difficult to artificially replicate such a complex and active market in a testing environment, he said.
As one solution, Isaacson and BATS advocates the use of test symbols by all exchanges in their production environments, which would allow broker-dealers to get hands-on experience with new algorithms or order types to see if any potential problems exist. Another possible solution, registering trading software, suggested by Lauer from Better Markets, could provide the SEC with an increased capacity to conduct real-time surveillance of the markets.
A “kill switch” procedure also could be used by exchanges to shut down aberrant activity. Executives of all four major national exchange operators – BATS, NYSE Euronext, Direct Edge and Nasdaq – as well as the Financial Industry Regulatory Authority said Friday they are prepared to set up such procedures, to head off a repeat of a Knight-like incident.
Citadel’s Nazareli said a kill switch would be an important tool at the exchanges, which are in the primary position to halt any problems early on.
“We don’t want a problem at a single firm to become a market problem,” he said.
Citadel has its own product, Fusebox, which sits outside its trading system and can act as a sort of kill-switch.
The Fusebox halts trading when Citadel’s systems get overloaded, similar to the way an overloaded fusebox in a home works.
Want to Fix a System? Break It First
Want to see what happens to your trading desk or matching engines when you shut down a part of the electronic systems that support them? Try it.
That, in effect, is what the U.S. equities industry’s post-trade processing utility, the Depository Trust and Clearing Corp., does. At the Securities and Exchange Commission’s Market Technology Roundtable Tuesday, DTCC managing director and chief development officer Albert Gambale detailed a tactic his firm routinely uses to test its systems.
The clearer selects a branch office and has it halt operations and stand-down to see how the rest of the company’s nationwide system handles the shut-down.
Such testing of how well a system can keep working without one of its parts in place could have helped, for instance, Knight Capital on August 1 when it unleashed its flood of erroneous orders onto stock exchanges. Shutting down the misfiring part, in theory, would not necessarily have meant shutting down all its market making operations.
Crisis preparation such as these shutdowns and the tests that go with them are important because in the moment of crisis responsive actions must be handled with “military precision,” according to Saro Jahani, Chief Information Officer of Direct Edge, operators of the EDGA and EDGX exchanges. “Often, there is no time to find the absolute best person to handle it—rather, you have to focus on doing the right thing for the market.”
Direct Edge has its own experience with this type of crisis. Last October, the SEC cited Direct Edge for two system failures that allegedly caused millions of dollars in trading losses.
The first incident occurred in November 2010 when untested computer code changes caused the two exchanges to overfill orders. The unwanted trades involved an estimated 27 million shares in about 1,000 stocks, totaling roughly $773 million.
A second incident occurred in April 2011 when an exchange employee accidentally disabled database connections and disrupted the exchange’s ability to process orders and cancellations. EDGX members filed claims for more than $668,000 in losses.
Following the SEC’s citation, Direct Edge undertook “a comprehensive plan” of regulatory compliance, which included significant investments in technology and personnel, to harden its systems. Indeed, yesterday Direct Edge partnered with software tester Coverity, Inc., to improve how it tests software before it gets used by its exchanges. The “Edgile’’ approach automatically tests code and identifies issues as code is being written.
Dr. M. Lynne Markus, Professor of Information and Process Management at Bentley University in Waltham, Mass., outlined four basic elements that exchanges and trading firms should follow, to prevent glitches or limit their impact.
First, reduce complexity of systems, so there are fewer steps and fewer opportunities for errors, Prof. Markus explained.
Second, test systems thoroughly, to make sure they work on their own and in conjunction with systems with which they interact.
Third, monitor activity, at all times, so difficulties or aberrant actions are spotted instantly.
Fourth, keep backup capacity in place, to take over trading or speed recovery from a shutdown, so normal trading can resume as soon as possible.
“To ensure a robust system, these four strategies have to be addressed and utilized together,” explained Prof. Markus. “Otherwise, I am afraid that the crises we’ve seen over the past year or so may become the new normal.”
Communication is also key in managing a crisis. Anna Ewing, Executive Vice President and Chief Information Officer of NASDAQ OMX, said that during any market event, her firm has teams to address direct outreach to both regulators and customers, often while continually updating the exchange’s status alerts.
Throughout a crisis, it is essential that an exchange communicate with these groups—something that cannot be left to computers or software, she added. “In this, people matter,” Ewing said. “This is a part of the culture where people’s common sense is needed.”
Nasdaq OMX has set aside $62 million to compensate its customers for problems experience in handling orders in its IPO Cross system on the day Facebook shares went public; and in transferring those orders to the Nasdaq Stock Market itself.
Experts on the SEC roundtable also recommended segregating key duties so that any single dependency on any one piece of software or one person can be avoided.
They also recommended identifying whether problems are internal or external to begin with. And then having responses planned out, where possible.
“It’s not only about identifying the trigger points that lead to problems, but you have to identify what customers need during this crisis and how those obligations can be fulfilled,” said David Bloom, Head of UBS Group Technology at UBS.
Of course, fixing trading software systems during a market malfunction has its own particular problems.
“In most cases, you’d still be trying to triage it,” explained Lou Steinberg, Chief Technology Officer at TD Ameritrade, the online brokerage. “At that moment, you may not know what is happening and whether to fix it, or try to leverage the capacity you need to recover from it.”
Kill Switches and Best Practices Are Coming
The implementation of so-called “kill switches” and a uniform best practices guide to electronic trading stand as the two high points of the Securities and Exchange Commission’s Market Technology Roundtable held on Tuesday.
That’s the opinion of two of the roundtable panelists. Jeff Bell and Chad Cook, chief executive officer and chief technology officer, respectively, at Lime Brokerage, told Traders Magazine advanced talks surrounding the implementation of kill switches are under way.
The two panel discussions covered a plethora of topics, ranging from instituting so-called “kill switches” to the exchanges’ views on technology testing and how to possibly regulate the heady pace of technology deployment. Still, it was the ability to shut down trading that garnered the most attention.
“The need for kill switches and their implementation, as well as the need for some sort of standardized implementation of best practices were the takeaways today,” Bell said.
“Kill switches” refer to a device or technology that would allow a market participant to shut down trading due to a technical glitch, such as a malfunctioning algorithm. Many in the industry believe such a technology or failsafe switch could have prevented market events, such as the recent Knight Capital debacle.
While there was no time frame for instituting or activating the market failsafe kill switch under discussion, Bell added that all panelists, including representatives from the exchanges, regard such technology as a priority.
“When kill switches were discussed, there was a definite sense of urgency in the air and lots of interest and talk of the exchanges and industry coming together to work on this,” Bell said. However, no one elaborated on the money, manpower or technology necessary to put such a failsafe mechanism in place, he added.
Cook added that kill switches need to be at multiple trading locations with clear communications between them in order to be effective.
“Overall, there’s still a lot of work to be done on all fronts,” Cook said, referring to both kill switches and a best practices guide.
Despite the hurdles in implementing such technology, kill switches are coming, Bell said.
“Today the SEC got a green light to go ahead and adopt the kill switch concept,” Bell said. “There was a general consensus that this is the best mechanism to protect the market.”
