The database is dead. Long live ESP! With the explosion in algorithmic trading has come a voracious appetite for low latency data and a need for a speedy way to query that data. But the old ways of data management-querying a database built by a Sybase or a Microsoft-are ineffective when processing thousands of pieces of information every second. So now, sophisticated broker-dealers and high-end money managers have latched onto a new data management technology that allows their trading systems to make decisions in the blink of an eye.
The technology is known as event stream processing (ESP).
ESP is just one of the terms to describe technology that enables applications-and people-to monitor, analyze and act upon multiple streams of fast-moving, real-time event data. It is also known as complex event processing and data stream processing, and essentially means that traders and applications can detect patterns and intelligently act upon opportunities and threats in real-time.
Spending on ESP is set to rise exponentially, according to a report by Tower Group. The Boston-based research house is predicting outlays on ESP third-party solutions will be $67 million in 2006. Tower expects that spending to explode to $600 million in 2010.
Good Thing
At the forefront of the ESP surge is Apama, a British vendor with a next-generation algorithmic trading platform and ESP solution now owned by Progress Software. Apama was a pioneer in developing a platform to support the new kinds of trading at a time before most people knew what algorithmic trading was. Other vendors in this space include FlexTrade, Portware, Streambase, and newcomer Coral8.
Tom Price, senior analyst for securities and capital markets at Tower Group, said ESP solutions vendors are onto a good thing. "Data is proliferating and coming in at speeds too fast for human intelligence to interpret and translate," Price said.
"It almost becomes a requirement to have some kind of event stream processing engine with business rules, to monitor events in real-time that you deem to be crucial to your business."
The old-fashioned way of delivering market data-feeds, filters, phone lines and "green screens" or dedicated PCs-was transformed by digital market data distribution platforms in the 1980s; and for about 10 years this was just about as much as a customer could hope for in terms of data delivery.
Previously, databases had been used to store, index and query the data. The game changed in the 1990s. That's when automated trading systems, electronic trading platforms, algorithmic trading systems, and direct market feeds came along, the game changed.
Traditional methods of storing and querying data in databases are not suitable when data is changing as rapidly as it is today, according to Apama vice president and founder John Bates.
"After automated trading took off," Bates said, "this [traditional] infrastructure could not support the speed at which brokers and traders needed to get their data. This is where event stream processing came in."
The Apama platform was built after 10 years of research conducted by Bates and Giles Nelson, who were both at Cambridge University in England at the time (Bates was lecturing in computer science there). The company was founded in 1999.
"With quantities of real-time data doubling year on year," Bates said, "the research team that ultimately produced the basis for Apama had a vision: to change the way in which data was analyzed."
Bates' research team took a new approach, using event processing. The team was able to formulate a way in which events' flow through an engine, with that engine then extracting relevant data as defined by the user. This approach turned the notion of data storage on its head.
"We designed it for data flow. It's like panning for gold in a fast flowing stream, and the gold jumps into your pan," he explains.
The founders made a decision to focus on capital markets, said Bates, and this is how Apama became involved in algorithmic trading: "Now it happens we've become the main experts in it."
Apama's customer base is evenly split between the sellside and buyside, with hedge funds now the fastest growing customer segment. As little as 18 months ago, Bates said, Apama's customers were mostly sellside.
"Hedge funds now also want to have a competitive advantage from developing strategies of their own, for their own trading purposes."
Spreading
But business is also spreading through large investment banks, he says. "We often start with equities and move on through to futures and options within the bank," he said. Major investment bank clients include: JPMorgan Chase, which deployed Progress Apama's ESP within its global futures & options trading operation; Deutsche Bank, which uses Apama's Algorithmic Trading Platform based on a FIX architecture for program trading and arbitrage. And ABN AMRO, which chose the Apama Algorithmic Trading Platform to enhance the program trading desk. Apama began looking in earnest at algorithmic trading in 2002 when a sellside customer approached the firm: "At the time the buyside did algorithmic trading through brokers' pre-built VWAP-style systems. Our customer desired to go beyond that, leveraging its own intellectual property and strategies." Progress Software acquired Apama in April 2005. Bates said Apama's relationship with Progress helps what was formerly a small London and Cambridge-based company to scale its presence in 80 countries with global sales teams in major centers such as New York, London, Tokyo and Hong Kong.
Responding to Data
The original vision, though, Bates said, remains the same: to help companies with new requirements for the way they look at data.
"Traders need to see streaming ticks and quotes, exchanges and liquidity pools. They need to analyze and respond to patterns in the data in order to gain competitive advantage," Bates said.
Out of this was born a new product line: Progress Apama's Algorithmic Trading Platform, with Event Modeler module, allows clients to leverage packaged and proprietary algorithmic trading strategies. The Event Stream Processing platform monitors event-oriented data and applies sets of analytic rules to those events in real-time. It then determines the appropriate action to be taken in response. Event Manager is the real-time correlation engine for both platforms.
The Apama Dashboard Studio was launched in May, and offers a real-time visualization tool for the Apama ESP platform. Before this incarnation of the dashboard, some users had a fully embedded system where text would appear on the users' screens, said Bates.
The Dashboard Studio sharpens Apama's competitive edge against firms such as Portware and FlexTrade, as well as Tibco, webMethods and other Business Activity Monitoring (BAM) solutions firms, said Bates.
Ary Khatchikian, co-founder and president of automated portfolio trading software provider Portware, said that Apama competes with Portware on the event-driven engine side, but that Portware takes that to the next level.
"In addition to the event-driven technology, we offer trade management, position management, portfolio management, and trade strategy on our front end," said Khatchikian.
Portware and FlexTrade also compete with Apama on the algorithmic trading platform side, said Bates.
No Coding
Khatchikian said: "Although Portware started as a trading system for hedge funds upon which they could write their own algorithms from the ground up, it now offers pre-packaged algorithms that clients can use as a template and then customize."
Using Portware's front end, users can change their algorithms on the fly in real-time, without having to have coding expertise, says Khatchikian. Portware also offers a customizable exchange simulator, where users can back-test algorithms.
FlexTrade said it offers a more than just multi-asset class algorithmic trading and event processing.
Anita Karppi, vice president of marketing for FlexTrade Systems, said: "We offer a very extensive front end GUI that allows traders or portfolio managers to make adjustments to strategies on the fly. Additionally our rules engine has an extensive multi-asset blotter, a FIX engine, a smart router, a pre- and post-trade analysis system and several robust APIs for integration with all phases of the investment process (OMS, compliance, portfolio accounting, etc)."
Karppi said that, along with advanced pre-packaged trading functionalities, FlexTrade also provides out-of-the-box' algorithms.
Not Selling Algos
Bates cautions that Apama's reputation on the algorithmic trading platform front may be overshadowing the strides it has made in other areas: "We do not position ourselves as an algorithmic trading vendor," Bates pointed out. "We provide templates for customers to build their own algorithms. They will still have their own quants, but even with them, it takes weeks or months to roll out new algorithms, and it is about moving quickly."
Competition is getting fierce as the ESP trend takes off. There are other firms trying to penetrate the ESP technology side-such as Streambase and start-up Coral8-as well as Tibco and webMethods, said Bates.
Bill Hobbib, vice president of marketing at Streambase, said the firm operates on a different principle than Apama. Streambase normalizes incoming data, monitors its latency, enriches it for analysis or algorithmic trading, and feeds it into applications such as position or order management software, compliance systems, algorithmic or automated trading systems, and risk management solutions.
"We developed a platform that extends SQL, a familiar language that clients can build applications on top of. It is not just for algorithmic trading, but also for trade-cost analysis, real-time compliance and portfolio updating," said Hobbib. "We are orders of magnitude faster than the competition," said Hobbib, "And our biggest competitor is custom coders."
Streambase's customer base spans large investment banks to hedge funds, which use it for processing and querying market data, as part of the customized infrastructure for algorithmic trading or for smart routing. NYSE Arca (formerly ArcaEx) uses Streambase as part of its electronic trading infrastructure. Despite the competitive pressures, Bates does not appear worried. The Progress deal has given his start-up some muscle. And the electronic trading boom is just beginning.
"If you define algorithmic trading as automating what traders have traditionally done manually," he said, "then there is a massive future for us."
