Commentary

Elaine Wah

Modern Markets, Modern Metrics - A Blog By IEX

In this blog by IEX's Elaine Wah, the newest public exchange looks to refute public claims that the metrics it uses are designed to inflate its own volume numbers and mislead people.

Traders Poll

Do you think it's a good idea to conduct an access fee pilot to assess the pricing models used by many trading venues?

Yes

67%

No

0%

Should have had a pilot program a long time ago.

33%

Free Site Registration

August 9, 2006

Making Decisions in a Blink

By Melanie Wold

Also in this article

  • Making Decisions in a Blink

The database is dead. Long live ESP! With the explosion in algorithmic trading has come a voracious appetite for low latency data and a need for a speedy way to query that data. But the old ways of data management-querying a database built by a Sybase or a Microsoft-are ineffective when processing thousands of pieces of information every second. So now, sophisticated broker-dealers and high-end money managers have latched onto a new data management technology that allows their trading systems to make decisions in the blink of an eye.

The technology is known as event stream processing (ESP).

ESP is just one of the terms to describe technology that enables applications-and people-to monitor, analyze and act upon multiple streams of fast-moving, real-time event data. It is also known as complex event processing and data stream processing, and essentially means that traders and applications can detect patterns and intelligently act upon opportunities and threats in real-time.

Spending on ESP is set to rise exponentially, according to a report by Tower Group. The Boston-based research house is predicting outlays on ESP third-party solutions will be $67 million in 2006. Tower expects that spending to explode to $600 million in 2010.

Good Thing

At the forefront of the ESP surge is Apama, a British vendor with a next-generation algorithmic trading platform and ESP solution now owned by Progress Software. Apama was a pioneer in developing a platform to support the new kinds of trading at a time before most people knew what algorithmic trading was. Other vendors in this space include FlexTrade, Portware, Streambase, and newcomer Coral8.

Tom Price, senior analyst for securities and capital markets at Tower Group, said ESP solutions vendors are onto a good thing. "Data is proliferating and coming in at speeds too fast for human intelligence to interpret and translate," Price said.

"It almost becomes a requirement to have some kind of event stream processing engine with business rules, to monitor events in real-time that you deem to be crucial to your business."

The old-fashioned way of delivering market data-feeds, filters, phone lines and "green screens" or dedicated PCs-was transformed by digital market data distribution platforms in the 1980s; and for about 10 years this was just about as much as a customer could hope for in terms of data delivery.

Previously, databases had been used to store, index and query the data. The game changed in the 1990s. That's when automated trading systems, electronic trading platforms, algorithmic trading systems, and direct market feeds came along, the game changed.

Traditional methods of storing and querying data in databases are not suitable when data is changing as rapidly as it is today, according to Apama vice president and founder John Bates.

"After automated trading took off," Bates said, "this [traditional] infrastructure could not support the speed at which brokers and traders needed to get their data. This is where event stream processing came in."

The Apama platform was built after 10 years of research conducted by Bates and Giles Nelson, who were both at Cambridge University in England at the time (Bates was lecturing in computer science there). The company was founded in 1999.