In a filing with the Securities and Exchange Commission on market stability, the Financial Information Forum said 25% of firms do not test their systems against ‘worst case’ scenarios and 25% only test new software releases.
The other 50% of firms conduct ‘worst case analysis’ on all new software as well as periodically on installed software, the group’s Market Stability Working Group found.
The working group was created after the Securities and Exchange Commission announced that it planned to hold a Market Technology Roundtable, in the wake of the near-destruction of Knight Capital Group. That market maker nearly went bankrupt after releasing a flood of erroneous orders onto stock markets on August 1.
The forum’s filing was submitted to the SEC on November 1, the day after stock markets re-opened after the two-day shutdown caused by Hurricane Sandy. The participants in the forum include institutional broker-dealers, full-service broker-dealers, trading and back office service bureaus, and market data vendors, as well as exchanges, electronic communications networks and clearing firms.
Questions in FIF’s survey “were framed around who determines the different scenarios a robust system must be able to comply with and which personnel and their back-ups get involved during a crisis situation,’’ according to program director Arsalan Shahid.
When market participants plan for enhancements that increase bandwidth or handle additional message traffic, the “most useful” types of analysis cover the effect on system load, effect on bandwidth and effect on latency. Identifying impact, through worst-case scenarios, and simulating responses, through fire drills, were lower priorities, the forum found.
The most widely used tactics for identifying problems as they happen and address them are application-specific monitoring systems, used by nearly 90% of firms responding. Roughly three out of four firms use cross-application monitoring systems, while three out of five use incident management practices. Only about one in three use complex event processing software, in the mix.
Roughly 40% roll back software, if an error is identified in a production system. But only 6% to have used a kill switch in production.
To fix its “large software bug,’’ Knight Capital rolled back its trading software, to an earlier version. Since the event, national exchanges have said they are prepared to install ‘kill switches’ that would shut down trading, when order flow gets out of hand.
