In November, Merrill Lynch contracted with Wombat Financial Software to provide a market data infrastructure for the broker-dealer's next-generation electronic trading platform. The initiative underscores a trend underway at major trading houses: Grappling with the large quantities of real-time market data demanded by algorithmic and automated trading systems and other applications. Electronic trading groups at broker-dealers, proprietary trading firms, hedge funds and other sophisticated players are beefing up their market data infrastructures and eliminating as much latency, or delay, as possible in the quest for speed.
Vendors such as Wombat are pouncing on the opportunities they see. "A firm can't trade without knowing exactly where the market is and the shelf life of market data is steadily dwindling," notes Ken Barnes, Wombat's vice president for business and planning. "Proper market data infrastructure provides that price discovery in a matter of milliseconds, while the data is still indicative of the actual market."
Low latency is Wombat's calling card. That's because slow market data-data that's just a few hundred milliseconds tardier than the next guy's-translates into slow trading decisions. And slow trades are lost opportunities in the world of automated trading. The tick data fueling trading decisions also feeds other applications such as pre-trade analytics and real-time risk management programs.
The primary elements of a market data infrastructure are feed handlers, ticker plants, a messaging
platform that transmits data between systems and integration software.
Electronic trading groups and prop traders increasingly need direct exchange feeds instead of consolidated market data feeds provided by data vendors such as Reuters and Bloomberg. With volumes increasing, they must also continually build, enhance and customize their market data infrastructure to efficiently process and distribute that data internally. Many organizations have multiple infrastructures, data feeds, feed handlers, databases and applications to get the data where it needs to go in the right format-all roped together with middleware and other integration tools.
Quick and Quicker
Everyone wants the same thing: speed. "All data has the same essential characteristic-it's constantly updating and customers want to get it as quickly as they can with the lowest possible overhead and latency," says Mark Mahowald, president of 29West, a provider of high-speed messaging software for financial market data.
Data volumes have multiplied as a result of decimalization and the continuing need to hide orders by breaking them down in size. Volumes are set to rocket even higher as the New York Stock Exchange sees more electronic trading in a Regulation NMS environment. That, in turn, will drive more electronic trading, algorithmic and other high-frequency types of trading, which will further boost volume. New data products from market centers will also push volumes higher.
According to the Financial Information Forum, a centralized information bureau for U.S. equities and options market data run by the Securities Industry Automation Corp., message traffic has surged in recent years. In November the sustained 1-minute peak for market data was 121,000 messages per second, up 116 percent from the 56,000 messages-per-second rate in November 2004. As the peak rate increases, the ceiling for market data traffic must also rise to ensure there's enough headroom for sudden spikes in data.
OPRA, the Options Price Reporting Authority, which provides options quote and trade data from the six U.S. options exchanges, said last August that entities getting direct OPRA feeds must be able to handle peak rates of 173,000 messages per second and 1.3 billion messages per day by this summer. That's up from 53,000 messages per second at the end of 2004. Options constitute two-thirds of the volume tracked by the Financial Information Forum.
Market-making firms with auto-quoting capabilities need data with near-zero latency. If another firm gets the data faster, the market-maker's bids will be hit and offers lifted before they can be updated.
The Needs of Algos
Algorithmic platforms also have the same need. "If a trader is trading manually, a few hundred milliseconds doesn't matter," says Vijay Kedia, president of FlexTrade Systems, a vendor of an algorithmic trading platform. "But for algorithmic trading, latency is as important an issue as the data itself."
A year ago, FlexTrade got its feeds through commercial market data providers, except for ECN books, which it got from the source. Now all of its feeds come directly from the source-from the New York Stock Exchange, Nasdaq and the ECNs. FlexTrade aggregates the feeds and provides that to customers.
Reuters, Comstock and other data consolidators have worked with trading firms to reduce data latency in their feeds, but the fact remains that more hops mean additional latency. "Anyone who gets data straight from the source finds an immediate shortcut," Kedia says. The consolidators now support a handful of direct feeds from exchanges.
DRW Trading, a large proprietary trading and market-making firm in Chicago that operates mostly in the fixed-income markets, has been getting direct data feeds from the two Chicago futures exchanges for a long time-seven years. But now, to meet the needs of just a handful of equities traders, DRW plans to switch from its Comstock consolidated data feed for equities data to direct exchange feeds. "Speed is a pure requirement," says Tony Verga, the firm's director of infrastructure.
Some banks and brokerages pull in the feeds themselves and write their own feed handlers to read and extract the necessary information from the streaming data. Others buy feed handlers from vendors such as Wombat, HyperFeed Technologies and InfoDyne Corp.
Besides direct feeds, trading firms are paying more attention to their internal messaging platforms. The need again is for low-latency data distribution. Messaging platforms move data rapidly between systems and applications, usually based on a publish/subscribe model that allows applications to get only the market data they need instead of an entire stream of data.
Firms that do a lot of algorithmic trading and that republish their data for use in pre-trade and real-time risk management models put a bigger load on their transport and messaging infrastructure.
Software firm Tibco has been the leader in this area. Rendezvous, its original messaging product designed for large-scale distributed application environments, is the dominant messaging platform used by banks and broker-dealers. The Palo Alto, Calif.-based firm also has two other messaging products.
Three-year-old 29West positions itself as the David to Tibco's Goliath. The Chicago-area company claims it has a higher-performance messaging product than Tibco's RV. (Tibco didn't respond to a phone call by press time.) Mahowald notes that in tests the LBM product (which stands for latency busters messaging) can handle 1.4 million messages per second between two simple dual CPU Dell boxes.
In 2004 Wombat licensed 29West's LBM product as a messaging platform that can be used in its market data infrastructure alongside or as an alternative to Tibco's RV product. 29West has also licensed its LBM product to a number of large banks, exchanges and prop trading outfits. Mahowald anticipates signing another 30-40 direct clients this year.
On the Prowl
But while prop trading firms and other automated trading operations are always on the lookout for newer, faster technology, the adoption of new technologies is rarely a simple process. "The licensing fee is a small percentage of the overall cost of vendor-provided technology," says the head of an algorithmic trading group at a large broker-dealer in New York. "The real cost is connecting to all the applications and the rest of the technology infrastructure."
Mahowald, for example, knows his firm isn't about to displace Tibco. "If I'm Bank of America or someone with a worldwide presence, I'm not going to swap out my messaging overnight," he says. "I may buy a new messaging product for a block of 500 CPUs in a prop trading area, see how that works, and then gradually creep it out across the organization."
Trading Simulations
With data volumes expanding, firms also need bigger and faster tick databases so they can test strategies and simulate trades. "Years ago historical data was used primarily for day-end reporting. Now it's necessary for program trading," says Simon Garland, the Zurich-based chief technology officer at Kx Systems, a Palo Alto, Calif.-based software firm.
"If you're doing prop trading, you need to compare data from the last five seconds to data from the last five minutes and the last five hours. Ideally, you'd also compare it to data from the last five months," he says. Kx offers a high-speed, high-performance database for both real-time and historical data, which reduces latency. Vendors often sell separate databases for real-time and historical data, which greatly increases latency when testing data strategies on the fly. Kx has large database installations in more than a dozen global banks, as well as in a number of hedge funds.
Complex algorithms are now analyzing vast quantities of streaming market data and executing strategies in multiple asset classes, instead of just equities. There's also increasing buyside demand for "white-box" algorithms-user-customized algorithms that are determined and controlled by traders, rather than remaining opaque to them-since they allow traders to test and trade ideas immediately.
Greater computing power and the need to analyze streaming data have fueled the rise of a new technology called complex event processing. This refers to technology that can read vast streams of incoming market data and detect patterns that can be acted upon by an algorithmic trading system.
"This is usually deployed in-house by a proprietary trading firm for rapid analysis of incoming data streams, particularly to detect changes across multiple streams or in single streams within different time windows," says Mary Knox, research director in the investment services research area at Gartner Inc., an IT consulting and research firm based in Stamford, Conn. Some of the companies with offerings in this area are Progress Software Corp. (which acquired Apama Inc. last year for its event processing technology), Vhayu Technologies and StreamBase Systems.
John Bates, a Progress vice president in charge of Apama products, notes the rise in event processing technology and computing power means that automated trading firms can incorporate every tick and quote update into their analytics and trading strategies. Indeed, they must do so to stay competitive.
The expanding streams of market data must be processed and analyzed fast-even before they hit a real-time database. "Instead of storing data and then querying it, we let an organization specify the strategy upfront in terms of when/then rules," Bates says. For example, the strategy may be that when the spread between Microsoft and Oracle exceeds a certain level in the streaming data, Apama's algorithmic trading system is alerted to buy Microsoft and sell Oracle. "A trader can index the rules and stream the real-time data through them and the relevant data will stick onto the relevant rules and actions can be taken immediately," he explains. He compares this to panning for gold in a fast-flowing stream-with the relevant gold nuggets jumping into the pan of their own accord.
Reeling in the Big One
The Apama platform is installed at JPMorgan, Deutsche Bank, ABN Amro and other banks. Last year Apama began selling directly to buyside firms. One large fish it reeled in was Aspect Capital, a $2.4 billion hedge fund based in London, which currently uses the platform to trade foreign exchange.
So how fast can prop desks execute strategies based on streaming market data? The most sophisticated trading firms are competing on milliseconds-if not microseconds-and they're improving their models continuously. "The faster you pull in data, the faster you validate the data, the faster you optimize your portfolio, the faster you get to market-that's your edge," says an equities prop trader at a bank in New York.
But with everyone competing on speed, that advantage won't last. "That gives you a competitive edge for a while," says Gartner's Knox. "But at some point you hit the speed of light and things just don't get any faster."
At that point the competition may shift to making improved trading decisions. "If a firm is not competing on sheer speed, then it's competing on what it does with the market data information and the kinds of filters or decisioning rules that are applied to it," Knox says.
The New York-based prop trader recognizes this. He expects that by the time speed is no longer a differentiating factor, his desk will have moved on to other products and models where the quality of the market data is what's most critical. That includes the ability of models to recognize ever-more-complicated patterns in the marketplace, including patterns across asset classes and those that aren't necessarily fleeting. The all-out race for speed also raises a concern that expanding market data volumes could affect data quality. With more data processed and pushed out to applications, the reliability of that data-from redundancies to dropped ticks-could mean the difference between a strategy that succeeds and one that fails.
FlexTrade's Kedia notes that information overload is also potentially a concern. A prop desk may not need to get the price of Microsoft 100 times per second. "In response to the growing amount of data, applications will try to be smarter about what's necessary information vs. noise or non-usable information," Kedia says. "You can push out more data, but more isn't always better." But more data does still appear to be better. Then again, the fire hose of data certainly will grow larger.
FIX Gets Fast
FIX Protocol Ltd., in an effort to deal with the exploding volume of market data, released its FAST (for FIX Adapted for STreaming) protocol. This data-compaction methodology, which enables huge volumes of market data to be disseminated with low bandwidth requirements and low latency, already has the support of a number of exchanges. Their promised adoption of FAST, sources say, could be the first step in creating a standard market data interface for direct exchange feeds.
