The blink of an eye is an eternity.
At least it is these days on Wall Street. It takes 400 milliseconds, or four-tenths of a second, to blink–and that’s far too long for an increasing number of traders. Black box traders, direct market access traders and algorithmic traders are all in a race to beat the other guy. And the best way to do that is to get their hands on the market data first and their orders to the exchanges first. They expect to do it in microseconds or, at least, single-digit milliseconds.
To accommodate today’s demand for speed and the accompanying surge in message traffic, market centers and brokerages are spending millions to upgrade their infrastructures. Tabb Group last year estimated the industry was spending about $100 million a year on low-latency technology. It predicted that figure would reach $170 million by 2010. New hardware and software are replacing older technology. Co-location facilities are sprouting up. Consolidated feeds are being snubbed for direct feeds. Code is being rewritten.
At the same time, a slew of vendors and service providers have mushroomed in the past five years offering high-speed software and connectivity to traders and venues alike. Firms like Options IT, FTEN and Lime Brokerage command a significant share of the exchange access business. Messaging middleware vendors such as 29West are replacing incumbents such as TIBCO. Blazingly fast FIX engines from the likes of Rapid Addition are pushing aside traditional boxes. The upshot is a wholesale revamping of the market’s infrastructure.
"The Street is in a race to zero latency," Michael Lynch, Bank of America Merrill Lynch’s head of Americas equity execution services, told Traders Magazine. "It’s very important. There is no question about it."
The drive to reduce communication delays on Wall Street is at least as old as the telegraph, but the push to microseconds–one equals a millionth of a second–is the latest chapter of the story. For the past several years, speed has been measured in milliseconds, or thousandths of a second. But with improvements in hardware and software–notably Intel’s Nehalem chip, multi-core CPUs, InfiniBand linkages, Microsoft’s .NET framework, bigger network pipes and specialty messaging software–faster speeds have become possible.
Behind the drive to reduce latency are two groups: direct market access traders, such as statistical arbitrage shops, and the algorithm-wielding electronic trading departments of the major brokerages. The former, often of the high-frequency variety, are considered the more obsessive of the two, but the electronic trading desks are equally vigilant.
"DMA clients tend to be more focused on their latency," Merrill’s Lynch said. "Our algorithmic trading and private clients tend not to be as focused about getting the lowest latency, but we are focused in getting it for them."
Speed of Light
Exchanges and brokers typically say the improvements they are making to throughput and latency benefit all of their customers. But for exchanges, at least, it is the high-frequency traders who are in the drivers’ seat. That’s simply because they do most of the trading these days. Tabb Group estimates high-frequency traders account for 70 percent of the share volume.
Zero latency is impossible, given the barriers to achieving the speed of light, but the delays are likely to shrink further, sources say. Some in the industry, including NYSE Technologies, are already making their latency measurements in nanoseconds. One nanosecond equals a billionth of a second.
Latency can be defined as the time it takes information to move from Point A to Point B. The information could be a quote traveling from an exchange’s data feed to a broker’s trading application. Or it could be an order traveling from a broker’s trading application to an exchange’s matching engine.
Transmission is not instantaneous, of course. Sent out in the form of packets, the data encounters three types of latency. It must be converted to bits (transmission latency), shuttled across a network (propagation delay) and processed by trading and middleware applications (processing latency).
The greatest latency can occur on the network, especially if the journey covers a long distance. Like a person in a taxi bound for the airport, a packet can get tied up in traffic. Just as the traveler faces stoplights and tollbooths, the packet may hit catch points in the form of routers and switches. Thousands of packets can queue up at these catch points, slowing progress for all.
The next big source of delay occurs within an exchange or broker’s infrastructure. In the brokerage, for instance, feed handlers take in raw data and convert it to usable data. It is then transmitted by messaging middleware to a trading application such as an algorithm. Some middleware configurations, especially those based on older technology, can add unacceptable levels of latency.
The bottlenecks frustrate traders, especially those of the high-frequency variety. These shops, which may do thousands of trades per day, base their strategies on faster access to market data and trading venues.
Their obsession with speed is legendary. Take the practice of co-location, for instance. Co-location involves placing one’s trading server next to a market center’s servers at a specially built facility. It is a popular way for high-frequency traders and others to eliminate those precious microseconds by reducing the physical distance between their servers and the exchanges’. One story making the rounds is of a high-frequency trader who visited his firm’s co-location facility and measured the length of the cable connecting his server to the trading venue’s. Apparently, one foot of cabling equals one nanosecond of latency.
"The exchanges are being pushed harder and harder by that set of market participants who want to trade at really high speeds, and that would be the high-frequency trading groups," said Donal Byrne, chief executive of Corvil, a firm that monitors and manages latency for various industry players. "And, of course, the exchanges compete amongst themselves to attract that order flow."
Vicious Cycle
The increase in speed brings with it an accompanying increase in messages. The faster a trader can suck in the data, the more trading he is able to do. And the more trading he can do, the more quotes, orders, reports and cancellations he generates. It becomes a vicious cycle. That’s why throughput, the ability of systems to move not just one message but millions of messages very quickly, represents the other side of the coin. Large brokers and exchanges do not invest just for speed’s sake. They must be able to keep up with ever-increasing volumes.
In 2004, messages from options and equities trading peaked at 60,000 per second. In 2008, that figure was 987,000 messages per second.
No one is unaffected by the drive to microseconds. Market centers have been under pressure at least since BATS Exchange went live in 2006. BATS, established to cater to high-frequency traders, boasts turnaround times of less than 400 microseconds. Suppliers of market data and message transport technology such as Thomson Reuters and Bloomberg have had to grapple with a slew of upstarts offering speedy direct feeds to latency-sensitive broker-dealers and hedge funds. Firms that build software have had to hire programmers to rewrite their code to make the applications more efficient. Financial extranet providers such as BT Radianz and TNS have watched as specialty firms built faster networks to connect brokerages with market centers.
Two Milliseconds
If the industry needed proof that trading had passed into a new era, it needed to look no further than the New York Stock Exchange this summer. The subsidiary of NYSE Euronext scrapped its 33-year-old SuperDOT platform order delivery and processing system, as well as an internal routing system called Post Support System. In its place, the NYSE installed its Super Display Book system, technology based on NYSE Arca’s trading engine. The move cut the time it takes to execute an order from 105 milliseconds to five milliseconds, according to the exchange. That’s down from 350 milliseconds in 2007. NYSE customers now get order and cancellation acknowledgements in two milliseconds, the NYSE added.
Five milliseconds is a far cry from BATS’ 400 microseconds, and NYSE executives acknowledge they still have work to do. Still, the rollout this summer of the Super Display Book system was the culmination of an 18-month project that saw the Big Board completely reengineer its underlying hardware and software architecture using technology it acquired with the purchases of Euronext, Wombat Financial Software and Archipelago. The NYSE completely replaced its order entry, order database and routing systems, market data systems and pieces of its post-trade system.
The move to revamp its dated infrastructure potentially opens doors that were previously shut to the NYSE. At least one bulge shop refused to send any of its algorithmic flow to the NYSE because it was too slow, an NYSE exec told Traders Magazine.
Keeping Up
The exchanges and alternative trading systems are not shy about touting their speeds. Chi-X Canada ATS announced recently it boosted its capacity to be able to handle 175,000 messages per second, a 500 percent increase from previous capacity of 30,000 messages per second. Chi-X Canada, owned by Instinet, has benchmarked its average response time for marketable immediate-or-cancel orders at about 350 microseconds. That’s at least 10 times faster than any other major Canadian market center, it contends. Previously, Chi-X Canada’s internal latency was pegged at 890 milliseconds.
Chi-X Canada, based on the same technology as Chi-X Europe, launched in February 2008. Tal Cohen, Chi-X Canada’s chief executive, said the ability to process and disseminate information quickly is important "no matter what business you’re in. We need to keep up with our customers." Cohen said part of Chi-X’s strategy to drive latency lower is to run the system on "commodity" hardware. That way, the ATS can simply plug in the newer and faster boxes once they become available. "The drive for latency is not slowing down anytime soon," Cohen said.
All things are relative, of course, and speed is no different. Market centers are forever engaged in a ferocious battle with each other to win market share. In their desire to impress traders, they will often put out overly rosy latency numbers, critics charge. "There is a lot of confusion, and, in some cases, obfuscation about the actual latency at the venues," Corvil’s Byrne said. "It is impossible to make apples-to-apples comparisons."
That applies to both the speed at which the venues disseminate market data as well as the speed at which they convert orders into trades, Byrne added. The problem is that the venues typically offer up an average number, which may be of little use. "If you measure it at one time, you get one number," Byrne said. "If you measure it a few seconds later, you get another." He believes trading venues should publish a schedule of numbers for all times during the trading day.
Doug Kittelsen, chief technology officer of execution management vendor FTEN, is also concerned. He believes exchanges should quote their latency data in the 95th percentile, or at the slow end of the spectrum, rather than the median or average, which is standard. "You want to see what the curve is like all the way through to the end," Kittelsen said, "not just the middle."
FTEN is part of a new breed of vendor that has grown up with the high-frequency traders. It partners with the prime brokerage arms of the big brokers to offer hedge funds low latency trading technology services, including market access. The arrangement saves the big brokers from having to make substantial investments in infrastructure and know-how. At the same time, it makes it possible for hedge funds to use multiple prime brokers, but flow their orders through a single trading infrastructure. FTEN offers market data feeds, connectivity to trading venues, co-location and risk monitoring. Hedge funds only need to plug their trading applications into FTEN’s APIs.
Executives of the firm say they give hedge funds "complete transparency" into their latency data at all times. FTEN publishes data when it receives an order, when the order is sent out, and when the exchange acknowledges the order. All this is done on a message-by-message basis, FTEN says.
Options IT offers a similar service. The British firm has had a presence in the U.S. for the past six years, but has stayed under the radar. It says it is in the "infrastructure services" business. It provides buyside and sellside black box trading houses with a soup-to-nuts low latency trading environment into which the stat arbs can drop their trading applications. Options IT takes in market data over direct feeds from market centers around the world and passes it through just one switch on the way to the client’s server. The servers themselves are connected by high-speed links known as InfiniBand and 10-Gigabit Ethernet, the latest high-speed linking technology. Options IT’s standard servers are based on Intel’s new Nehalem chips, the fastest in the world. It co-locates its servers with those of the exchanges, further reducing latency. Options IT claims it can transport a message across the Atlantic from the trader’s application to the gateway of an exchange in 34 milliseconds and between U.S. exchanges in microseconds.
Options IT’s chief executive, Nigel Kneafsey, estimates there are about 150 different components that need to be "optimized" for low latency trading. His firm built its own network which it says is faster than those of its competitors, the major extranet providers. The firm has 120 clients, a list that it says is starting to include the prop desks of the major investment banks. Because of Wall Street’s financial difficulties, some brokerages are reducing their investments in infrastructure and turning to firms such as Options IT for help. "We are seeing some of those prop desks and algorithmic trading units moving onto our platform," Kneafsey said, "instead of investing to effectively replicate what we have built."
Recently, BNP Paribas sold the co-location operation of its prime brokerage business to Options IT. The deal gave Options IT three new data centers, bringing its total to 15.
Mahwah, N.J.
Much of the speed advantage customers get with Options IT comes from its practice of co-locating its servers with those of exchanges. In Chicago, for instance, the vendor places its servers in a data center run by Equinix, the largest operator of co-location facilities, because that’s where the Chicago Mercantile Exchange houses its servers. More such facilities are on the horizon. What will become the largest co-location facility in the U.S. is being built by NYSE Euronext in Mahwah, N.J. The 400,000-square-foot facility is expected to open in about a year. About one-fifth of the space will be used by NYSE Euronext. The rest is earmarked for clients.
Demand for co-location is strong and supply is tight. At least one network provider moved an entire point of presence (POP), or switching facility, to please one of its best customers. TNS did this in Toronto, where electronic trading has grown considerably due to the growth of ECNs such as Chi-X and Alpha. "We ended up actually moving a POP to a location that was viewed by one of our key customers as a better place to be in terms of reducing latency," said Jim Parker, a TNS executive responsible for sales engineering and product development in the vendor’s financial services division. "We’ve done it. It’s operational."
Co-location is not for everyone at all times, though, TNS executives say. The vendor, which offers its customers a range of connectivity services, notes that many of its latency-sensitive customers opt to co-locate only with those venues where they trade the most. For others, they will settle for a high-speed T-1 line. In London, for example, "They’ll co-locate with say BATS or Chi-X for a direct connection but then connect into our cloud so they can get to all the other exchanges," said Alex Walker, a senior vice president and general manager in the financial services division of TNS.
TNS was reluctant to offer up its latency data, given the range of configurations, distances and routes it offers, but did note that a typical rate would come in at less than one millisecond.
With co-location and the advancements in high-speed communications links such as InfiniBand, some believe propagation latency and transmission latency have been neutralized. What remains to be tackled is processing latency, or the time it takes the software housed at exchanges, brokerages and hedge funds to process incoming data.
While some players, notably vendor Exegy, advocate a solution known as hardware acceleration that can perform computing functions faster than software, most of the activity to date has involved software.
Feed Handlers
On the market data front, a crowd of software vendors has emerged in recent years offering to speed up the processing of that data at brokerages, exchanges and money managers. Feed handlers, messaging middleware and so-called "event processing" technology have become integral at many of the largest shops. On the order-sending side of the equation, a new generation of FIX engines has emerged.
The move away from consolidated feeds and towards direct feeds by the more latency-sensitive players has created the need for a so-called "feed handler." A trader would purchase one of these for every datafeed he required. The systems capture the raw feeds and convert them into usable data for further processing.
In 2008, NYSE Euronext spent $200 million to acquire the leading vendor of feed handling technology, Wombat Financial Software. The firm has since been folded into NYSE Euronext’s vendor arm NYSE Technologies.
NYSE’s feed handlers are typically installed in one of two places. They can be located within firms where they distribute into a market data distribution system that, in turn, distributes to users. These arrangements vary in terms of how latency-sensitive they are. Or they can be situated in co-location facilities where they take in only as much data as the trading application needs. The emphasis here is on speed.
"In that world," said Conor Allen, NYSE Technologies’ head of research and development, "getting the data in and parsing it extremely quickly is important. Because it is how quickly you can see the change happen that determines how quickly you can react to it. The reality is, that order is only there for one person. So if you react faster, you fill the order."
NYSE Technologies recently released an upgraded version of the Wombat platform based on Intel’s Nehalem technology. It can reportedly achieve mean latencies of between 10 and 40 microseconds, depending on the type of memory used. It is "ideal for co-location servers," the company stated in a press release.
Four or five years ago, Allen said, low latency platforms were in the 1.5- to 2.0-millisecond range. Today, the average latency of the feed handler for the NYSE’s Open Book feed is five microseconds.
The feed handlers, of course, aren’t the only things getting faster. The exchanges are, too. Plus they have more capacity. Combine a fast feed with a fast feed handler, and "the applications can make more decisions per second," Allen said. "That translates into more orders per second, which means more market data per second. Which causes the rates to grow. It feeds on itself a little. As each part gets faster, the overall rates increase."
Another one of those parts of the industry’s infrastructure is the FIX engine. The FIX messaging protocol was originally designed 15 years ago to be used in the transmission of order information between the buyside trader and the sales trader. Today, FIX engines are used to communicate directly with market centers. That has made the speed of the FIX engine much more important.
"Years ago nobody noticed if one message arrived in one-tenth of a second; the next arrived in a second; the following arrived in tenth of a second; and the one after that in half a second," said Kevin Houstoun, co-founder and chairman of British FIX engine vendor Rapid Addition. "Human beings don’t notice these times. Now speed matters very much."
Houstoun built one of the first FIX engines while responsible for trading technology at the former investment bank Robert Fleming & Co. in London. The bank later gave away the software to whoever wanted it to help get FIX off the ground in Europe, Houstoun said. He later left Fleming for Salomon Brothers in London where he co-headed electronic trading under one of the inventors of FIX, Jim Lehman.
Rapid Addition’s FIX engine–named Cheetah–consistently achieves speeds of, on average, 10 microseconds with a standard deviation of about four microseconds, according to the company. FIX engine latency a couple of years ago, Houstoun says, was about 100 microseconds on average. He chalks up his success in reducing that number to the help he got from Intel and Microsoft whose technology he used. Along with Intel and some exchanges and large brokers, Rapid Addition is a partner in Intel’s Low Latency Lab, which helps firms find ways to reduce their overall latency.
Rapid Addition is still the new kid on the block, counting four Cheetah customers. The London Stock Exchange is one. A large investment bank is running a "proof-of-concept" test, Houstoun says. The technology is geared towards high-frequency traders, as well as large brokers and exchanges "that need to deal with a lot of flow coming in from a lot of places," Houstoun said. "Either you need speed for speed’s sake or you need throughput."
Market centers usually need both. Direct Edge ECN, for instance, has seen its share of trading volume shoot from about 5 percent a year ago to about 12 percent today. To deal with the increase in messages flowing through its systems and prepare for the future, the ECN chose to replace its messaging middleware, TIBCO. It chose faster technology from 29West, a relative newcomer to the business. DirectEdge says installation of 29West’s technology has produced a "dramatic reduction in overall system latency" and increased its throughput.
"It allows us to use persistent messaging without the penalty," said Steve Bonanno, DirectEdge’s chief technology officer. Persistent messaging is typically slower than non-persistent messaging, but offers guaranteed delivery. No messages are lost as they can be with non-persistent messaging. 29West’s technology eliminates that latency "penalty," Bonanno explained.
DirectEdge is now able to guarantee its customers an order response time–the amount of time it takes to acknowledge the receipt of an order–of 300 to 500 microseconds. That’s measured from the time the order enters DirectEdge’s gateway to the time the acknowledgment hits the gateway. Previously, response times of DirectEdge’s three trading platforms were in the 1.2- to 1.5-millisecond range.
Although DirectEdge needed the changeover to 29West primarily for throughput, it couldn’t ignore latency. "Speed is the toll you have to pay to be in this business," Bonanno said. "Being fast is a given. That has to be there."
And faster than the blink of an eye, of course.
(c) 2009 Traders Magazine and SourceMedia, Inc. All Rights Reserved.
http://www.tradersmagazine.com http://www.sourcemedia.com/
