Ingrid Tierens, co-head of the electronic trading "strats" group at Goldman Sachs Electronic Trading (GSET), spoke with Traders Magazine recently about new developments in the world of algorithmic and electronic trading. Among the topics discussed were: the next generation of algorithms, order transparency and algorithmic testing.
Traders Magazine: What new trends are you seeing in algorithms or in electronic trading?
Ingrid Tierens: I would say a couple of trends in the algo space are noteworthy. First, clients are asking for increased transparency as to what is happening with their electronic orders. We are getting more and more requests related to how we end up choosing one venue over another, how our order placement logic works, etc. We haven’t waited for clients to reach out to us –we’ve done this proactively and published a lot of research around the turn of the year on this. We are trying to provide a framework on how to think about order routing and to help bridge the knowledge gap by providing clients with metrics to help them think about how this all works. We go as far as developing client-tailored analyses. This has been a theme we’ve seen over the last 12 to 18 months.
TM: What else is on clients’ collective mind?
Tierens: A second theme in the electronic trading space is that even though at first glance the many providers in the electronic space may have similar suites of algorithm, there is the realization that not one size fits all. Our clients are diverse – the spectrum ranges from very quantitatively, shorter-term focused clients to customers that are fundamentally driven and have a much longer investment and trading horizon. The last thing we would advocate is that all these diverse clients trade the same way even if they have access to the same types of algorithms.
TM: How do you do this?
Tierens: We spent a lot of time with our clients to think about how we can fine tune the algorithms for their specific usage. There are a couple of different ways we go about it. First, we provide a broad range of post-trade execution quality analysis and consulting to clients. One obvious limitation we need to keep in mind when we work with clients is that we only have access to the flow they send through Goldman Sachs and can only do analysis based on what we’re seeing. However, to the extent that the client wants to partner up with us, we can delve deeper and potentially come up with ideas that might meaningfully impact the way they trade.
TM: Can you elaborate?
Tierens: Let’s say we have a client who runs several types of investment strategies running and different traders execute each strategy. When we look at all their executions in aggregate, put them into a big pot if you will, nothing interesting may jump out. However, once the client flags that the aggregate flow represents these different strategies and we are able to analyze them separately, we may come up with our advice on how to execute each strategy in a more optimal way. It might be because the PM for one strategy has a short-term momentum approach and another PM runs a long-term value fund.
One person in our group is fully dedicated to execution quality analysis on behalf of our clients. We’ve expanded our execution quality analysis quite a bit over the last one to two years. Two years ago a client might have been happy with a standard report, but now we end up doing more deep dives to really consult with our clients and I think they’re really interested in hearing what we have to say given these deeper examinations.
TM: Can you give an example of things you’re providing now that weren’t present in algorithmic execution quality analysis two years ago?
Tierens: We examine price dynamics around a client’s order flow. A couple of years back a typical report would mainly tell a client his shortfall relative to when the order originally came into the broker and whether the stock either moved up or down during the rest of the trading day. Now, we look at orders in much more granularity and examine price movements relative to the first execution, relative to the average execution and to the last execution. We check to see if there is momentum or reversal and if we are actually impacting the price by trading too aggressively. We are going into much more detail beyond what a client might have been satisfied with a few years back. Now, we drill much more deeply.
For those clients who are more fundamentally driven and enter into large positions in a specific name traded over multiple days, we’ll look at the analysis through a process we call "multi-day re-stitching." In addition to providing separate metrics for the portion of the trade done on day one, the portion of the trade done on day two etc., we re-stitch these trades over these multiple days and provide analysis benchmarked to the time the order came in on day one. In looking at trades in a re-stitched fashion versus an unstitched fashion, we may come up with very different conclusions. Clients don’t want to just look myopically at one single trading day but rather how their trading behavior is affecting the name they are trading over the full trading horizon.
TM: Anything else?
Tierens: In addition to incorporating a lot of additional dimensions in the traditional execution quality analysis as we just talked about, we go a step further and conduct "structured experiments" to fine-tune, to the extent possible, the appropriate settings for a specific algorithm. We conduct these structured experiments both internally and on behalf of external clients.
Years back in the electronic trading space, algorithm improvement was more a process of trial and error. Someone might notice something with the behavior of an algo, it may or may not have been doing something it was supposed to. Then they would put a "Band Aid" on it, make some changes and hope for the best. Now we have a much more scientific approach, which we call "structured experiments." It is where we will form a hypothesis, try to examine how the algo can be changed, then make the changes, measure the impact of the changes and potentially uncover unintended consequences. It’s not just making a change and hope it works out the right way.
TM: Can you expand on how "structured experiments" work?
Tierens: First, we implement this new idea of how to change the algorithm. Then we randomize the order flow into the algorithm and do a 50/50 experiment so we can compare the algo’s performance on an "apples to apples" basis. We have half of the orders go into the "old version" and the other half into the "new version" and let the experiment run over a time period that is appropriate for the amount of flow we receive in that algorithm. Then we analyze algo performance on an ongoing basis and once we have enough data, come up with conclusions. On the back of that, we’ll do a rigorous analysis we’ve called "COPE" – Child Order Performance Evaluation — where we can really break down the algo’s performance in all of its underlying components.
So instead of presenting only an overall number to measure performance, such as the new algo overall gave us "x" amount of basis points of improvement versus the old regime, we can now break down results and attribute them to very specific changes. We can for example look at the algo’s aggressive component, its dark and passive components, see how much is going to which one and measure how each component is performing relative to when a particular child order was created. This allows us to do a full attribution analysis and see where our performance improvements come from.
The process has come a really long way in terms of evaluating what’s going on within the algorithm and how it’s behaving. And because this is a much more analytical and structural approach, it gives us much more insight into how we can further develop the algorithms. This is a real sea shift in algorithmic development.
TM: How long do you conduct these structured experiments for?
Tierens: The testing period isn’t based on a time metric, but rather based on the number of orders that we get. If we’re working with clients, we always tell a client when we’re running an experiment that the more flow they send us the more conclusive the analysis gets. That way we get a bigger data set to compare against. If you look at comparisons, the bigger the sample the clearer the results become.
TM: Is there a minimum flow size that you like to see before making a change to an algorithm?
Tierens: We like to think of it not in terms of shares in an order that are executed but rather the total amount of orders we get. Depending on how elaborate the experiment is and how different the changes are to the algorithmic logic we’re thinking about, we like to see a minimum of several thousand to tens of thousands of orders. You’re not going to be able to evaluate a change based on a couple of hundred orders. We’ll look at many cuts of the data and what the order characteristics are across the two 50/50 buckets to make sure they are aligned before drawing any conclusions from the experiment.
TM: How do you provide more transparency into an algorithm’s logic or how it works?
Tierens: Obviously, this is an area where proprietary knowledge is at stake and where you won’t send algo code out to a client. Rather, we provide access to our algorithmic experts to clients who want to learn more about the algorithm. And it is easiest to do this within the context of their particular flow. In the most extreme case, where we will be running a structured experiment and doing the pre- and post-type of analysis we discussed earlier, we will walk them through why we think they should be going one way instead of another.
We want to make sure clients feel comfortable and have a level of understanding of what the algorithms are doing. Not all clients are the same–so we have to be able to satisfy their individual transparency needs and tailor our approach to their needs and objectives. The way we’re set up is that the algo developers in our team are not extracted from client interaction. We encourage and actively pursue direct interaction between the people who develop the algorithms and the end users. We come up with ideas because we are observing from a performance perspective but the client perspective is incredibly important as well. The structured experiments process is very interactive and at the end of day, the goal is that the client will wind up with a much better execution experience.
TM: How do you measure or monitor an execution venue’s toxicity after it executes a trade?
Tierens: We’ve identified a number of metrics to examine one venue versus another. One we’ve come up with is what we call the "bad fill ratio." It’s a measure of toxicity that looks after an execution happens whether or not the price of a stock is more or less often moving against us. Toxicity is only one dimension to compare venues. We also focus a lot on how fill rates differ across different venues. Both are quantitative numbers and we can compare them across venues and time and make routing decisions on the back of those comparisons. We analyze them on a regular basis and that helps us decide if we need to make changes to our routing logic.
