New Michael Lewis book on finance and high-frequency trading


Have to admit I still favor number 8 below over Lewis book. But I may be biased.

This is a great piece, and I absolutely agree with number 8. Everyone listen to Cliff Asness!

So far, I do too. However, I haven't read the book. Look for my review of "Flash Boys" in Advisor Perspectives later this spring! I may be asking you to provide a countervailing view or some background.

Michael Lewis is spreading misinformation about HFT. Telling average people the "market is rigged" is a terribly misleading and potentially damaging message.

HFT has virtually no relevance to the long term small investor (except perhaps to slightly lower the transaction costs.)

Lewis gives the example of Microsoft stock. Well, I've actually bought and sold Microsoft stock in the last few years. Go tomorrow and try to buy or sell 100 shares of Microsoft. Look at the published bids and asks for a few minutes, then submit your order. I will bet that the spreads will be small (probably just a penny) and you will get a price right at or within a penny or two from what you expected when you submitted your order. Thus your trading cost was, at most, a couple of dollars. Plus a brokerage commission that for me is under $10. This is *vastly* cheaper than it was a couple of decades ago for a "small investor" to trade stocks.

No, you probably can't make money as an at home day trader. But it was always thus. Just buy and hold (or better yet buy index funds) and don't worry about it.

The worst thing you can say about HFT is that it is wasteful competition between the trading firms. So I don't have anything against trying to fix that, but telling small investors the market is "rigged" because of HFT just makes me angry.

What are your feelings about machine readable news? Isn't that part of the problem - the algos are acting on news faster than humans? And the HFTs sometimes get the data feeds ahead of the regular humans. This is aside from the plain old frontrunning that the book is probably about.

No matter what the system is, there is one basic fact that's true about financial markets: Market moving events happen, and no matter what some participants will get to trade on those events before others. Let's say there were no direct exchange feeds, it doesn't matter HFT firms would still co-locate directly at the source of the consolidated feed. Humans take 150ms to blink an eye, that's an eternity to a CPU. Even if a human and a machine are using the same feed the machine can still process way faster than any human can. The direct feeds aren't about HFT getting a leg up on the regular humans, it's about getting a leg up on other HFT firms. Even before computers there were still human beings who stood on the floor of the exchange and made trades. These humans could respond to market moving events much faster than humans in Ohio.

There's a simple lesson here: don't try to play this game unless you have a compelling reason to think you can win. No, you're not going to trade on news stories faster than Wall Street firms. No, you're not going to become a millionaire day trading from your basement. But if you invest your money in, say an index fund, this doesn't matter to you, because you're not competing in these games. The market is certainly not a "level playing field", it never has been and it never will be.

Consider the extreme scenario where all or many of the "informed" participants left the markets. So all that's left is HFT's trading against each other & index funds or other such boring, portfolio balancing participants.

Wouldn't that be an iffy state to move to; would HFT's have as much of arbitrage making opportunity in such a market? What's the risk that HFT's make the game so one-sided that conventional position taking participants no longer have the appetite to do so?

The extreme scenario is not that informed participants leave the market, but that the uninformed ones do. The profits of informed traders are the losses of uninformed ones. If dumb money leaves the market, there are no profits to be had, and thus, there will be no trading. This is Akerloff's "Market for Lemons." As for the risk of this--if it hasn't happened already, why should it start now? Market participants were quite happy giving up far larger sums of money to the Nasdaq market makers and NYSE specialists of yesteryear. If they were the least bit interested in quantifying the value-add of their trading, they would have stopped gambling long ago.

Read Matt Levine for a good answer to your question.

+1 for the Matt Levine link. IMO he's the best financial journalist out there.

Is HFT mostly a zero sum game? Or not? Like, in an average month, is the profit of HFT firms that had a good month compensated by the losses of the HFT firms that had a bad month?

If not, where's the profit coming from: is it small investors stupidly trying to day trade, or non-HFT conventional traders?

In any market, at any given time there may be a surfeit buyers relative to sellers or vice versa. Markets clear by price adjusting until supply equals demand. Financial markets are unique in that there is a very high amount of information heterogeneity among the participants. I pretty much know how much limes are worth to me, if limes get too expensive I cut back on the caipirinhas. However as a typical investor I really don't know how much Microsoft is worth, if it goes up by $5 I should probably keep buying it as part of my portfolio regardless. I definitely don't know what it's worth down to the cent, so for very small price adjustments the vast majority of investors won't change their preferences.

As such financial markets typically have specialized traders commonly referred to as liquidity providers or market makers. Through various mechanisms they sit on the other side of the natural order flow of buys and sells that come into the market. When there are more natural buyers they fulfill the excess demand by selling for slightly higher prices, and vice versa. Without these participants prices would swing wildly since the vast majority of investors are very price insensitive relative to typical daily fluctuations. Going back to the information heterogeneity, this strategy is not without its risks. Sometimes supply and demand is out of balance for random order flow reasons, but sometimes its out of balance because an actor or actors with private information about the financial asset is accumulating a position. As a market maker you are naturally on the other side of these actors and are subject to adverse selection that can easily eat up all your profit.

One of the most important skills to being a liquidity provider is knowing when order flow is likely to be "informed" and to set prices accordingly. This entails the construction of quite complex models and careful analysis of historical data, and is another reason the role is fulfilled by highly specialized participants. The advantage of modern HFT versus earlier incarnations of liquidity providers is that computers are very good at analyzing huge amounts of data and quickly adjusting prices to changing order flow conditions. By raising the transaction costs for informed traders, they lower the transaction costs for uninformed traders who are more price sensitive. This brings more trading, particularly of the benign kind, into the market, which decreases the proportion of trading that's informed, further lowering transactions costs, all in a virtuous circle.

Think of the profits HFT firms make as equivalent to an underwriter in an insurance market. Assume the insurance market is highly plagued by risk pooling problems and its difficult to separate the high risk participants from the low risk ones. Much more of the high risk people sign up, driving up the costs and keeping many of the low risk people out. Assume new technology comes along that makes it much easier to identify a high or low risk person. Underwriters using this technology can offer much better rates to the low-risk, which is going to bring much more of them into the pool. That's going to produce a positive sum gain for the overall insurance market, and we'd expect the technology-equipped underwriters to be rewarded handsomely for that.

@HFT Trader

Yes, but what proportion of a typical HFT firm's volume is market making?

Also, when you say HFT raises "transaction costs for informed traders" how exactly do you mean? Can you elaborate?

I'd say 95% or more of HFT volume is related to providing liquidity. I don't know how you'd define "market making," but philosophically I would say providing liquidity means being willing to buy when the market naturally wants to sell and vice versa. This may take the form of providing two sided quotes all the way taking liquidity in the technical sense of crossing resting orders but trading in the opposite direction of recent order flow. Strategies where an algo is trying to parse the news and trade ahead of everyone else sound cool and get a lot of press, but make a very small proportion of HFT volume. News announcements happen pretty infrequently per stock, so there's simply not that many times you get to trade relative to providing liquidity.

What I mean by informed traders are people who's order flow has predictive values for the returns of a stock. The classical example is an insider trader. But it extends to any trader who earns consistent edge or alpha on his trades. The counterexample are index funds or passive investors who aren't buying for any informational reason but only to balance their portfolio. By contrast Warren Buffett may not even have access to any non-public information, but he has the mind of Warren Buffett, so the fact that he's buying a stock is very good news for the stock (and bad news for the counter-party he's trading against). Let's say you are quoting Microsoft at $35.25-$35.26, some of the trades will be uninformed and don't predict which direction Microsoft is going. On these you'll net in expectation $0.01 for a buy and a sell round trip. But sometimes the person trading against you does know where Microsoft is going, if that smart trader wants to buy he's going to sell to you at $35.26. It may start off looking like a good deal, but you'll now be short and the stock is probably going up. The proportion of how many informed traders to how many uninformed traders there are at any time determines the cost of the liquidity, or how wide or narrow the bid-ask spread gets set.

In fact liquidity providers are even subjected to the informed knowledge of their competitors even if they're not directly trading with them. Let's assume that both you and someone else are quoting Microsoft $35.25-$35.26, and say on any given trade you each have a 50/50 chance of being crossed depending on who got to the front of the queue. Let's say your competitor has an alpha that predicts which way the stock will tick next, if he's bearish he changes he shades his bid down to $35.24-$35.26 and shades his ask up to $35.25-$35.27. Now when someone wants to trade in the direction that he thinks Microsoft is going you only have a 50/50 chance of your quote getting filled, but a 100% chance of getting filled when someone's trading in the opposite, bad direction. The direction of the stock and direction of your inventory will have a negative correlation, and hence a negative impact on your trading profitability. You are still subjected to your competitors information even though you are not directly trading with him and only trading with uninformed flow.

The implication of all of this is that times when you think the market is loaded with information, means that you as a liquidity provider generally need to be compensated better for this, i.e. set a wider spread. For example during the morning hours spreads tend to be wider because more informed traders typically tend to trade then, responding to news and announcements that accumulated the previous night. During the afternoon spreads are narrower as much more of the trading tends to be uninformed portfolio rebalancing. Another example is if there's an unusual burst of trading, either in a specific stock or across the market, then that indicates a sudden introduction of information that needs to be incorporated into the market. There are many, many ways that HFT segments off informed trading from uninformed trading, and adjusts the costs of liquidity accordingly.

Since by definition informed traders are trading more heavily during the times when the market's loaded with information, raising the cost of liquidity during these times raises the average cost of liquidity for informed traders. That's counterbalanced by a lower cost of liquidity for uninformed traders. Better segmentation of informed order flow magnifies this effect. Overall that's a very good thing for the market, since uninformed traders are almost always much more price sensitive to liquidity than informed traders (since the latter have an edge to compensate them for trading costs), lowering transactions costs of the former at the expense of the latter brings more trading into the market and increases liquidity. Second the cost of liquidity is directly tied to the market impact of trades (wider spreads mean that each trade tends to move the market by a greater price increment). Raising the cost of liquidity for informed traders means that the market will respond more to their flow relative to the uninformed traders. Since the informed traders by definition know where the stock is going this increases the price efficiency of the market.

You HFT apologists missed the elephant in the room: the flash crash. HFT should not be banned, just taxed, to slow down the herd behavior. That goes for most stock trading. It's not socially useful so a tax is efficient.

Modern electronic markets have delivered drastically lower transactions costs, higher liquidity and more efficient prices since for the past 7 years. The May 5 flash crash constituted approximately 20 minutes of pathological behavior. That means 99.99664% of the time HFT electronic markets have been delivering excellent and consistent service. This is like someone suggesting we go back to Blockbuster because occasionally Netflix goes down.

Besides, the same behavior happened will before electronic trading. In 1987 traders simply stopped picking up the phones. The only difference is with modern HFT it took less than an hour for the market to recover, whereas in 1987 it took several weeks.

@HFT--so if the end of the world happens in only 20 minutes due to a worldwide exchange of nuclear weapons, we should allow nuclear weapons to be bought, sold, and used by everybody, since after all 20 minutes out of centuries is less than 0.001%?

As for your 1987 Crash analogy, that argues for further regulation of financial derivatives such as "portfolio insurance". Or at least more taxation of these new products.

You're likening the Flash Crash to the end of the world. Here's a challenge can you find a single major firm that went out of business due to the events of May 5, or any major fund that suffered a 10% or greater loss?

Just because two parties unrelated to you trade a $30 stock you own for $0.01 a few times it doesn't affect you in any way. It kind of looks scary and it makes good news, but it has no real economic impact on anyone. Like I said as far as I'm aware there were no serious repercussions of the Flash Crash on any major market participant.

Why does machine readable news affect the small long term investor? By the time I see it on the news and get over to trade on it, an hour is past and I am nowhere near the forefront. That, and I am a buy and hold index fund guy...

Ed: "The worst thing you can say about HFT is that it is wasteful competition between the trading firms. So I don’t have anything against trying to fix that, but telling small investors the market is “rigged” because of HFT just makes me angry."

I really want to fix the wasteful competition though! This is a lot of value creation of dubious utility that would be much better directed almost anywhere else. (How much better off are we as society that markets clear in .00001 seconds instead of 1 second? I claim that our benefit is zero.)

"How much better off are we as society that markets clear in .00001 seconds instead of 1 second? I claim that our benefit is zero."

A lot of people only see the value in sub-second trading as markets reaching equilibrium prices faster. They justifiably respond "So what?" if prices are stale for a second or two. But the impact goes far beyond that. The final price is informed by the price discovery process. Financial markets consist of many anonymous traders, some informed, some uninformed. The market has no bullet-proof way to disentangle the latter from the former. The presence of uninformed traders act as noise in the system. Their activity by definition has no ultimate information to contribute to the price, yet the market will always respond to their trades because of the non-zero probability that their trades may be from informed traders. And just the same the market will always under react to the activity of informed traders. This causes price formation to be less accurate than if everyone's information was publicly known to all participants.

As I've tried to convey in my other posts, a great deal of the effort of HFT ultimately boils down to segmenting informed from uninformed order flow. HFT's presence causes the market in aggregate to respond less to the uninformed traders and more to the informed traders. That reduces the noise inherent in market prices. So it's not just an issue of prices converging faster, but prices converging to more accurate levels. Without a doubt more efficient capital markets certainly do offer a substantial social benefit through the more efficient allocation of investment.

Your argument is tempting & elegant but is there a way to empirically test it. i.e. Do we have any empirical evidence that "prices converging to more accurate levels" happens better in a post-HFT world?

More to the point, is there even such a thing as an external, "accurate level" for a price outside of what is actually observed in the market?

In short, how do we test your hypothesis? Is it falsifiable?

Just to clarify I'm not saying there's one true objective price for an asset. Prices are ultimately manifestations of the market participants supply and demand schedule. But in markets where there is a high level of information asymmetry, like financial markets, you can talk about what price would clear if all private information was made public. (And to clarify private information includes private analysis of public information, e.g. Warren Buffet picking stocks). In the presence of private information the market price will drift around this hypothesis price, but always contain embedded noise.

The falsifiable question is an excellent one. It's certainly difficult. The first challenge is that if you had a way to quantify how inefficient a stock price was as you could probably trade on it and make money. It's also difficult because the HFT transformation went hand in hand with some major macroeconomic changes that certainly can easily outweigh any microstructure effects. I'll give two approaches I'd suggest, one simple, one hard. The simple one is to take to look at the autocorrelation of returns on some short horizon, days, hours, maybe even minutes. Auto-correlation is a simple test of how predictable a stock is given its prior information. Stocks that are perfectly efficient should have no predictability, ergo autocorrelation should approach zero. It's a well known fact that short-horizon autocorrelations have significantly declined over the past decade.

The second approach is much more complex and I honestly don't know what the answer would be. It might be a great project for an ambitious academic. Take a stock over some time period and slice it into some small unit of time. For each small unit of time divide the net shares traded by the change in price. That gives you an approximate estimate of market impact. Like I said before good market makers should increase the market impact during informed periods, and lower it during uninformed periods. Now average the market impact over a much longer period. Go back to each time slice and adjust the price change on the average market impact. The difference in the price over time should quantify how much market maker segmentation is adding to the price discovery process. You can of course compare this calculated error in the pre-HFT period to the post-HFT period.

HFT Trader, first of all thanks for your excellent contributions.

But I also don't understand the benefits of having market information incorporated in milliseconds rather than, say, tenths of seconds.

It seems to me that if we did something like go to synchronous trading at short intervals, or added random delays, or whatever, it might eliminate some wasteful competition for pure speed, like for example tunneling through the earth's crust to make a shorter fiber optic link. Do you see a downside too this?

(Again this has nothing at all to do with the market being rigged against small traders.)

I outlined some of my reservations about random delays in my response to Falkenstein one comment down. To quickly summarize my main hesitation is that large firms will simply pay for many trading entities or sessions. They'll enter the same order across each of their many sessions, and hence have a much higher chance of winning the random draw than a smaller firm with only one session. The wasteful competition to go faster will transform into competition to maintain as many sessions as possible.

Batching at very small intervals is not necessarily a bad idea, but it should be well less than a second. Probably around a millisecond or two, since the technological effort to respond at this speed is very trivial so almost all of the speed overhead would go away. Much longer than a millisecond blunts price discovery, particularly the equity markets tend to respond 3-10 ms after an index future move. Market makers need the ability to adjust their prices during these micro-events, or else they're subjected to much higher adverse selection. (As long as we're talking hypothetical I'd move the Chicago futures matching engine servers to New Jersey in the same locale as the equity exchanges. That would reduce the mountains of money that gets pored into the Chicago-New York data link.)

Furthermore I wouldn't determine the batch winner based on randomness. One it has the problems I mentioned above, and two it doesn't serve consumers. Rather I'd reduce tick sizes drastically so much more competition could happen at the price level. If there were multiple orders matching another order in the micro-batch rather than picking the winner based on randomness, the one with the best price would be selected. This turns most of the current competition for speed, or number of sessions in the random scenario, into a competition of price improvement. Consumers will realize the benefits of getting even better prices, and the competition won't be wasted in the sense that it is today.

(Though none of this implies that all or most HFT effort today is along the lines of wasted competition. Most HFT effort is directed towards the productive endeavors or improving price discovery and increasing liquidity, only a small proportion is dedicated towards winning the speed race).

Cliff: you need a better publicist so that you can present your thesis on 60 Minutes the week it's published. And evil bad guys.

The funny thing about the 60 Minutes piece that ran tonight was that the hero firm (IEX) that is fighting HFT has a silly speed bump. If speed per se is the problem, a speed bump just moves the advantage backward, faster is still btter. A better solution would be to have the exchange add a randomized speed bump, that would add, say, 1 to 100 milliseconds latency. That would turn a 10ms advantage into only a 60% higher chance of getting in front of its competition.

The key is competition. If a dark pool puts in rules that traders like, they will flock there. Further, you can take advantage of HFT by implementing vwap orders, because these firms compete and use HFT to give you the best price, so you are then indirectly using HFT tactics, not fighting them.

But if an investor is really are peeved by the tenths of a cent HFT make on their algorithms, they aren't long-term investors. The same people who want to throw sand in the gears with transaction taxes allege HFT are killing the stock market by their at most penny profit. Lastly, where was the outrage when Microsoft was quoted in 1/4s in the nineties?

Lewis has better hair. Ok, he has hair. PR people can only work with what they're given.

It sounded to me like they were selectively using the speed bumps to target HFT connections.

"If speed per se is the problem, a speed bump just moves the advantage backward, faster is still btter. A better solution would be to have the exchange add a randomized speed bump, that would add, say, 1 to 100 milliseconds latency. That would turn a 10ms advantage into only a 60% higher chance of getting in front of its competition"

A small randomized speed increment isn't necessarily a bad idea, but the numbers you're citing are way too long. Meaningful price formation occurs well below 100 ms. For example market makers will significantly widen quotes and raise the price of liquidity in the 3-10 ms following a change in the price of the index futures to segment out other HFT traders. This allows the average uninformed trader to realize tighter spreads, since they're not responding to the small window following market microstructure events. Plus it increases price efficiency and discovery by resulting in informed trades having larger market impact.

If you're going to randomize I would definitely recommend nothing more than 1ms. Any serious co-located trading firm is processing quotes and sending orders in well under 1ms, probably closer to 100us depending on how complex the strategy. Getting to this is a trivial technology cost, the serious technology cost is reducing processing time down to <50us. Keeping it under 1ms avoids most of the effect I mention above.

The final problem with randomization is already present on some exchanges, like the CME. Currently there are separate market network gateway, that traders are randomly assigned to. These gateways relay the orders to the matching engine. If one gateway becomes congested, it's possible that another participant at another less congested gateway might get his order matched first, even though his order arrived later. In some sense the CME already has randomization in place. But there's massive discontent with the system, and the CME's made a big investment to make the system a pure first-in first-out one. The problem is that larger firms will simply pay for many sessions, whereas smaller firms only have one or a small number of sessions. From a randomization it favors the larger firms because they can place the same order across many different sessions and hence are more likely to win the race. Like buying many lottery tickets, instead of one. Unless your randomization system deals with the problem I don't see how it can be successful.

You seem to know what you are talking about, so here's a layman question: It seems to me, that for the HFT traders, the underlying asset is mostly irrelevant; the asset just provides a convenient game, or puzzle, or gamble for the smart guys to peg their wits against, and the smarter guy (or lower latency hardware, or faster algo.) gets the rewards.

Now, if so, shouldn't that be a disincentive for the rest of us, who hold on to these assets because of some perceived inherent value? Say, I bought cows that I think will yield the most milk over their lifetimes, & then a large band of gamblers started to gamble using said cows (for maybe cow racing, cow fighting or something) doesn't that detract from the utility for people actually relying on the cows as a milking asset?

Don't you think there might be brokers who buy and sell cows, who will buy a cow from you even if you don't personally know anyone who wants a cow right now? And that this service would be valuable in providing a somewhat liquid market for cows? Do you someone could make money doing this? Don't you think that if you have more information about the cow market, and can react faster to price changes and supply and demand shifts for different kinds of cows, that you will be more successful?

Somehow, HFT seems different from a conventional trader / brokerage. HFT doesn't care about the asset itself outside of its purpose as a convenient way to compete on algorithmic cleverness against other HFTs.

Maybe I'm wrong.

HFT algos don't know anything about the companies they trade in the same way that that Netflix doesn't know anything about the movies its recommending. Netflix's algo can conclude that people that like Biodome probably also like Ace Ventura. The computer will never understand the subtle ties between the comedic geniuses of Pauly Shore and Jim Carrey. Netflix is simply looking at the aggregated behavior of millions of viewers. Similarly an HFT algo can look at the aggregated order flow of millions of trades occurring in the market and come to the conclusion that the auto sector is worth more relative to the market than it was ten minutes ago. It doesn't know why, but the individuals making the trades that it's looking at have valid reasons of their own. The HFT algo is simply interpreting their disparate behavior and individual opinions in the same way that Netflix is with their viewers.

More to the point new fundamental data is released quite scarcely. You only get corporate filings once every quarter, and you might get a new research report maybe once a month. Yet stock prices change second to second. Most of the time new information that's incorporated into prices doesn't come in the form of new fundamental information. It comes in the form of anonymous traders expressing their private information and private interpretations of public information through trading on the market. It's the anonymous order flow that HFTs have to focus on, not the fundamentals of the company, because it's that order flow that's affecting prices second to second, which is the horizon they're trading on.

@HFT Trader

No, that wasn't my point. I'll try restating: There's two different motives here, those looking for long term value deriving out of inherent quality of an asset versus those looking for a purely convenient instrument of speculation to be able to make money by virtue of the superiority of their algorithms or hardware.

To a HFT it hardly matters whether a stock is solid or junk, just how it can predict & exploit volatility, correlation quirks, or catch other patterns, or arbitrage fractions of a cent between markets, or very short horizon artifacts etc. faster than the fastest competitor playing the same game.

My question is, for someone looking for long term value from an asset shouldn't it be worrying if a mass of others is using the same asset for an orthogonal motive, with no relation to the asset's value.


I see what you're saying. I have two responses to this. First if you're worried about orthogonal market participants HFT is a drop in the hat. HFT makes a large percentage of the volume, but this is driven by small positions that turnover very quickly. For example the paper linked below looks at HFT activity in the S&P E-mini index future market. It finds that the average max inventory size of the largest type of HFT firms average about 800 contracts. In that group there were about 14 firms. So even if all the major HFT firms in that market all took their max positions in a single direction it would only constitute 0.4% of the 2.7 million contracts of today's open interest on the contract. To meaningfully push the price of a security up or down you have to trade in size relative to the market, HFT doesn't trade in size it just trades frequently.

Compare this in contrast to the CTA industry, which exceeds $2 trillion dollars. They are certainly buying or selling E-mini futures for orthogonal purposes (like based on a 200 day moving average crossover), in much larger sizes than HFT ever does. Or consider all the options traders who are delta hedging their books, they are definitely trading in very large size for very orthogonal reasons to buy-and-hold investors. Or billions invested in stat arb or momentum based funds. What I'm saying is that if this is a concern, HFT should be nowhere near the top of your list.

But the second point I would make is that why is it a bad thing to have orthogonal traders? It seems like a good thing to me, in the same way that having diversity in an ecosystem tends to make it more robust. If every trader in a security had the same or similar motivations as you then that means that they're going to tend to buy when you want to buy and sell when you want to sell. That means you'll probably be buying at the top and selling at the bottom, and also the volatility and price swings will be higher. I like traders who are orthogonal to me, because they're likely to constitute good counter parties to balance out when traders like me decide we want to buy or sell.


Can you articulate why it would be? Anyway, that's not really what HFT do, they are mostly just market makers.

Rahul, you seem to be missing the point. Average investors who just decide one day to buy MSFT or other stock are more likely to find a liquid market, with lower transaction costs, subsidized by the pro investors.

The cow example may be better if some 4-H kids snuck into the barn at night and had cow-brushing contests. The cows still give milk, the farmer has no clue; maybe they even give a fraction more milk because they are happy to be brushed.

I don't know squat about trading, but I do know that I'd love to be the firm making a penny on every single trade that goes down. Loss to me when I buy my pathetic few shares? A few cents. Loss to the fund holding my pension? Potentially considerable. But I'm glad that it doesn't bother you.

Go with your first instinct.

There's another type of firm making a good amount of money on every single trade. They're called brokerages. In fact they must be even worse than HFT firms and market makers because they amount they make is fixed and they literally never lose money on any single trade. The same goes for the exchanges. Now hold your hats on for this one, there's a whole list of firms, Visa, American Express, Paypal, and many others, that make money on every single trade that occurs in their network. In fact these aren't even financial transactions, but simple consumer transactions, and they take way more than a penny per trade.

Are you seriously saying all non-zero transaction costs are a type of injustice or theft?

Yes, HFT is much less outrageous than the credit card exchange fees. They are fixed by a quasi-monopoly. Whereas the cut that goes to the market makers for providing liquidity is not fixed but is subject to being competed away by other liquidity providers.

But do HFTs have a responsibility to make a market the way traditional market makers did? One reason put forth to explain the flash crash was that the algos would have suffered huge losses if they kept running, so they turned off the machines.

Obligating market makers to always make a market is not a free lunch. All obligations take the the form of forcing market makers to quote within some increment of the average price over some rolling period. (If you didn't give them price obligations they could simply quote offer to buy at $0.01 and sell at infinity, effectively pulling their quotes). By definition these obligated quotes are uneconomic. Since making markets is freely competitive forcing them to participate in some activity where they lose money will raise their prices in normal times to compensate for the loss.

As such market maker obligations almost always benefit sophisticated traders who can monitor their portfolios and sell out of their positions at the obligated quotes when the market is crashing. This subsidy is paid for by less sophisticated, primarily retail, investors who don't aren't likely to utilize the obligated quotes when they're available. If "stable markets" are a very important goal of yours then this might be worth it, but make no mistake market maker obligations are most definitely a tax on retail investors for the benefit of banks and hedge funds. (Plus I would suggest that circuit breakers are a better approach than obligations regardless.)

Except for Amex, which is also an acquirer and an issuer, none of the networks get over 1% of the transaction for domestic transactions. It's split by the acquirers, issuers, processors and networks. Second the networks seem to provide sufficient value that they can convince a third party (merchants) to pay them. If a merchant didn't want to pay the fees, they could stop accepting the network.

Obviously, some spread needs to be earned in order to make the market happen. We want this to be as small as possible without impairing the ability to make the market.

HFT claims that it does this, but I'm not sold. There are many reasons why spreads came down over the last two decades, to claim HFT is responsible for all that seems false to me. I wouldn't even rank it as an important cause.

Ultimately, an HFT trader doesn't increase the pie (no real world goods are being produced by his work, and I don't think Joe Schmoe puts more money into his 401k because HFT traders exist). So his only claim to being useful is that, even though his goal is to take as much as the pie as possible, that his activities mean Joe Schmoe somehow ends up with more of the pie then he did before because enough was taken from some third party that even after transaction costs (the primary cost being top STEM talent not inventing real world things) both the HFT trader and Joe Schmoe ended up better off. Moreover, you have to prove that is true for the marginal HFT shop.

As for liquidity we've all seen that being able to do MORE!, FASTER! hasn't exactly lead to stable financial markets over the last 15 years. And a bunch of algos run by a bunch of rootless nerds doesn't seem like the kind of thing that is going to provide liquidity when it really matters, like during crisis. Does Joe Schmore really need a whole lot of liquidity to buy his 100 shares of MSFT on a typical boring day. I have a feeling that order is going to get filled just fine without HFT.

From the other thread.

“But most of the time, markets are very tight – much much tighter than they were in the past when market making was a monopoly run by specialists.”

“You are (or should be) thrilled to be able to buy a thousand shares of MSFT with one transaction at a cost of half a cent a share (20 years ago, 12.5 cents a share, 25 times higher).”

20 years ago a lot of things were different. Most of those differences aren’t a result of a HFT shop getting a slightly faster internet connection. Can you really show that if your HFT firm shut down that spreads would be worse for Joe Schmoe? Especially in light of the fact that your goal is to make as much money off anyone (including Joe) as possible. I think this is a hard case to prove, especially for the marginal HFT shop.

“It’s OK for you to be unconvinced. But if we start trying to regulate it then the burden of proof needs to go the other way. In a free country things should be allowed unless there is a reason to ban them, not the other way around.”

Not really. Finance, most especially finance specifically divorced from facilitating real world productive investment, tends to be a zero (or negative with transaction cost) sum game. We aren’t talking about some consumer product or a factory getting built. No wealth is created by HFT. The only claim that HFT can make is that maybe somehow they make less of a spread then some world where they don’t exist (even then, no new wealth, your just claiming that the distribution leaves the little guy with a bit more of the pie). HFT, and things like it, take in a lot of very rare and scarce technical and intellectual talent. The best you can claim is that they are putting all that talent towards a zero sum game that maybe, through some really hazy assumptions that can be questioned, results in that zero sum pie possibly getting a little more into the common guys 401k, and only because the HFT trader failed at doing his job the best (taking everything possible would be best for his P&L). Considering that economic growth is mostly a function of such people coming up with real world technological breakthroughs that result in productivity growth (whose slowdown we have seen in very hard data for a long time) is HFT really something that should receive the benefit of the doubt.

Go with your first instinct.

Here is some more information on the content of Lewis's book.

So far, the HFT defenders in this thread have offered assertions ("Michael Lewis is spreading misinformation about HFT"), sarcasm ("Go with your first instinct"), attempts to change the subject (credit card fees are worse than HFT!), and the Office Space defense ("it's just fractions of a penny for the average guy") but little in the way of responding to Mr. Katsuyama's observation that HFT firms made it impossible for RBC to have its buy orders filled at market prices. When I want to order 100,000 shares of stock, how is it "providing liquidity" when I can only order 60,000 at the prevailing market prices and need to pay a higher amount for the other 40,000, only milliseconds later? Sounds like the opposite of liquidity to me, at least from RBC's perspective. If HFT provides liquidity, then it must be that RBC's solution to their order-filling problem, which cut out HFT, reduced liquidity. I'd be interested in seeing the HFT defenders explaining how this is true.

Anyone who needs to trade 100,000 shares of stock in an instant is either A) in possession of serious market moving information or B) a maniac. Since maniacs don't usually last long in positions where they're in charge of hundred of millions of dollars (as you would have to be to trade in this size regularly), it's probably A). It makes no sense for market makers to show that much on the touch when the only reason anyone would trade at such size is to try to exploit market moving information, which by definition makes it a bad trade for the market makers. Instead market makers today tend to show very tight spreads, at smaller sizes on the touch, that they refill regularly absent unexpected large order flows indicative of market moving information. Such a system is great if you're a small or patient investor, but means you pay more if you have inside information that you need to trade on immediately,

The liquidity you want in the market is the ability for uninformed traders (which includes nearly all of retail investors, index funds, etc.) to pay low transaction costs and easily be able to trade in the sizes that they regularly trade. These traders, never, repeat never, trade 100,000 shares in an instant. They might trade 100,000 shares in an hour, which if you're patient do a simple VWAP execution algo you will find the market will fill you tomorrow a cost that has never been lower in financial history. But if you're a sophisticated hedge fund, bank prop desk or some other shark of a trader who needs to execute 100,000 almost immediately because you have private information but quickly expiring information or alpha that needs to be seized upon immediately, you're out of luck.

You can't have your cake and eat it too. Uninformed mom-and-pop investors can get better liquidity and lower transaction costs, but that only comes at the expense of filtering and identifying highly sophisticated traders who are trying to pick off the market makers. Of you can give the sharks the ability to trade huge size at low cost, but that means the market makers are going to lose a lot of money on those trades. Since market makers aren't in the business for charitable reasons they'll have to be compensated somewhere else. That somewhere else will come in the form of higher transaction costs for uninformed, mostly retail, investors.

As far as I can tell Katsuyama also wasn't in the business for charitable reasons. Presumably he got high up in RBC because the trades he was making tended to make a lot of money. Meaning that his counter parties on the trade, the market makers, were losing money on those trades. Katsuyama seems to feel entitled to those market makers always being willing to lose money on his trades in the size he was used. But they didn't, the market got more efficient, and hence his strategy became less profitable. This happens all the time in trading, strategies that produced outsized and excess returns eventually get discovered by more people until they stop working. The only difference is most traders don't go crying to some pop-finance author about the "injustice" of it.

Thanks very much for the reply. I'm sorry that I picked "100,000" as an example number of shares. Perhaps I should have picked 10,000 or 1,000.

From this thread I've gained an appreciation that price elasticities are not imbedded in or implied by the "market price", and that HFT serves in some sense to allow for rapid "discovery" of price elasticity when large orders come in.

Curt, I'm an extremely small time investor, but often even in recognizable names the # of shares on the bid or the ask at any one time might only be in the 100s of shares. This bounces around frequently, but a market order to buy or sell even 1000 shares probably would push the price in many names because there's often less than that amount willing to be sold at that "ask" or bought at the quoted "buy" price a specific point in time. I think many folks use "limit" orders specifically for this reason - and they're a must for many mid to small cap names.

So are you saying you agree with Michael Lewis that the markets are "rigged" and it hurts ordinary investors, and you evidence is that some guy couldn't buy 100,000 (!!!!) shares instantly at "the market price?"

Seriously? Seriously?

Read HFT Trader's excellent comments.

To be clear, I'm not claiming that HFT isn't wasteful in some ways, or that it might not be improved, I'm just saying that

1. it hardly matters to long term small investors, with small orders where front-running is not a concern
2. if anything it lowers their transaction costs, which are historically very low
3. telling these people the market is rigged is irresponsible

Hats off to HFT Trader to taking the time for these detailed and thoughtful comments.

I keep reading Rahul and thinking he's falling into a zero-sum trap. Why can't investors benefit AND some HFT firms make money? Isn't there value creation here?

Yes, sure that can be too. I'm only trying to figure out if it is so.

What possible "right" do you have to have 100,000 filled? You "want to" buy 100,000. That is information. An efficient market does not let you buy all you want at the current price. I am sorry if you don't like sarcasm but it's quicker and easier than explaining the basic to every knucklehead.

That was to Curt F.

Sorry I haven't mastered responding on the right thread.

This blog is too high frequency for me, needs taxes and circuit breakers.

The "everyone is too stupid to understand what's going on" defense is not persuasive.

Stand alone it rarely would be.

Which side is making this argument defending what?

Oh that's precious!

If I understood a word of your posts 'm sure I'd be annoyed.

Answer direct questions or don't respond.

Wow, fans of HFT are up in arms over this book. What a surprise.

Whatever snarky remark you might come up with (and I'm personally not pro-HFT either), it's clear that posters like @HFT Trader seem to know a lot about how HFT works. So why not try to learn?

Whether they are HFT fans or not, their comments have substance.

There's not much to learn. They figured out how to game the system. That's all this is and all it was ever intended to be. Read the book The Money Game. Fifty years ago smart people saw these clowns coming.

And these smart people did nothing about it for fifty years?

A telling point.

And braying fools say inane things anonymously,

Keep it up "Z"

And petulant thieves get their panties in a twist when they are unmasked as nothing more than grifters. You d-bags are not the first guys to scam the system and you will not be the last.

I don't do HFT but nice try.

I am petulant but not a thief, and petulant as ignorant people making anonymous accusations try my patience.

HFT makes the little guy, and me, and most who trade better off, but ignorant Luddite anonymous ranters like to shout "d-bag" like they made a point and then act like their day had a purpose.

@HFT: here's what I don't understand. Are you claiming that HFT firms perform the same function as market makers? If so, do you announce yourselves as such? Otherwise, it appears that HFT is just a form of privately designed tax, that takes a small piece of lots of transactions, simply because they have figured out how to do so.

I don't think the analogy with credit card companies and brokers is correct - those entities perform a clearly defined service for the money they receive. I think one can argue that they are paid too much, but everyone knows who the players are and what they are getting for their money. It's not clear to me that this is true for HFT firms. They seem to me more like a tapeworm that inhabits the colon of the financial system - extracting a little nourishment from everything that passes by, without killing the host.

How would you feel if, when you went to the supermarket, and saw something you liked on the shelf, and extended your hand to put it in your cart, somebody suddenly jumped out of thin air, bought the item you were reaching for, marked the price up a little bit, and then sold it to you at the higher price? Is that a transaction you would have willingly entered into?

I think that your dismissal of the flash crash is a mistake. Clearly something went very, very wrong there. HFT increases the complexity of an important part of our economy in ways that are not predictable. Do we really want to be closer to the black swan event than we need to be? Especially when HFT doesn't seem to add any value to the system.

I like Katsuyama's solution to the problem: set up an exchange where HFT doesn't work. It's a market based alternative. If the people who buy and sell stocks would rather not submit themselves to a private tax, then they will flock to that exchange and avoid it. If they don't care, they can continue with the status quo.

"Are you claiming that HFT firms perform the same function as market makers? If so, do you announce yourselves as such?"

Yes, absolutely. 95% of HFT is in the business of providing liquidity in one way or another, which is the raison d'être of market makers. I don't know what you mean by "announce yourselves", if there was some special way to announce do this I'm not aware of it. Most modern exchanges have no designated market maker (which is a good thing because it lowers barriers to competitions, by allowing anyone to make markets).

"How would you feel if, when you went to the supermarket, and saw something you liked on the shelf, and extended your hand to put it in your cart, somebody suddenly jumped out of thin air, bought the item you were reaching for, marked the price up a little bit, and then sold it to you at the higher price?"

I don't know why you think this is how the market works. You seem to imply that HFT gets to see your orders before they are matched against the book. This is unequivocally not true. When you enter an order into the market it remains absolutely anonymous until the exchange atomically matches it, crossing any marketable counter-orders and giving you first time priority on the unmarketable portion. There is no way short of illegal hacking where someone can see your order and trade in front of you based on that knowledge.

Now it's possible that someone may place an order slightly before you (not knowing your order of course) and access that liquidity before you get the chance. But that's just life. Just the same a store can announce a limited availability sale and be sold out before you get there. I don't see how that's an injustice. But I assure you that unless you constitute a handful of small number of traders that have definitive alpha, that it's quite the opposite. HFT almost certainly wants to make sure that your order gets filled, not take it away from you. That's why HFT firms pay retail brokerages huge amounts of money just to trade against their clients and provide them with liquidity. The people facing some form of the problem you're alluding to are universally highly sophisticated traders, who are big boys and have no shortage of their own countermeasures.

"I like Katsuyama’s solution to the problem: set up an exchange where HFT doesn’t work. It’s a market based alternative."

Good, I'm all for diverse competition to ferret out the best approach. But here's my simple prediction: IEX will get a lot of publicity like the 60 Minutes piece, but fail to gain any significant market share whatsoever. The people behind IEX and their supporters will not admit their mistake, declare HFT to actually be beneficial and go home. Instead they will come up with some excuse about how the system is unfair or rigged and that stopped IEX from gaining any traction. But time will tell.

Wikipedia says,

Market makers provide a required amount of liquidity to the security's market. In return, the specialist is granted various informational and trade execution advantages.

Assuming Wikipedia is right, what are these advantages & don't they give some advantages to HFT firms playing market makers?

I'm primarily a futures guy, so take what I say with a grain of salt. You're referring to the NYSE specialist program (NASDAQ also has a market maker program, but as far as I understand the current obligations and privileges are mostly inconsequential enough to render the program moot). Out of the four major exchanges (NYSE, NASDAQ, DirectEdge and BATS), NYSE is the only one that maintains a floor trading system. As far as I understand the specialist program is a way to subsidize the floor traders and prevent all trading from going electronic. NYSE is generally viewed as the least transparent, and at times crooked, of the major exchanges because of its rather byzantine system. (Which is why its market share has generally fallen since 2007).

I know among major HFT firms some have a specialist presence, others have determined that specialists are not worth the cost and effort. Many strategies avoid trading on NYSE because it's not fully electronic. Having specialist designation does confer some advantage, but in the scheme of the entire US equity market its pretty tiny.

I am an equities guy and can confirm that what you say is largely true. However, being a NYSE DMM is a valuable privilege--the exchange's priority system means designated market makers effectively step in front of public order flow.

Cool, thanks for the clarification.

Thanks for the answers. Further comments:

- My (possibly obsolete) understanding of the function of market makers is that they provided an inventory of shares of a particular company, that they would make available to a buyer if no sellers were selling, and vice versa: being willing to buy when there was a seller of the company that they specialized in and there was no other buyer. They traded in both directions when nobody else would. This implies a willingness to acquire and hold inventory in a stock. Does an HFT do this? What is the longest you hold a stock?

- Further to that point, participants in the market knew who the market makers were for particular stocks. Do people know to knock on the door of an HFT firm to do business in a particular stock? I

- The NY Times article describes Katsuyama seeing stocks offered on the exchange at a price, and his act of agreeing to buy them at the offered price triggering an intervention by third parties that changed the price before delivery. He did not ask for this intervention. The seller did not advertise that some of the offered stocks would be delivered at a different price. This sounds wrong to me, and I'll guess that it would sound wrong to 99 out of 100 people. It sounds as though third parties, presumably HFT traders, are jumping into the middle of a contract, uninvited, and profiting from it. It's very hard for a layman to see how this is a legitimate way of doing business.

- So far, I find the technical detail you have contributed to be interesting, but the alleged value that HFT delivers to be unconvincing. Markets were liquid for a very long time before there was HFT. Automation is great until the algorithms fail, nobody can understand why, and nobody knows how to fix the system. If you are so certain of the benefits of HFT trading, and would like us to evaluate the information that you have provided on its merits, then identify yourself or your company. Stand behind your product. Otherwise it's hard to tell whether you are correct or merely good at rationalizing the practice.

"This implies a willingness to acquire and hold inventory in a stock. Does an HFT do this? What is the longest you hold a stock?"

By their very nature anyone who quotes a tight two sided market is willing to to buy or sell at an equivalent or better price than anyone else. So yes HFT absolutely acquires inventory. You may be operating off descriptions of strategies that execute pure latency arbs of buying in one market then almost immediately selling in another. This constitutes a small proportion of activity. If you refer to the CFTC study I cited above it found that the typical major HFT inventory size throughout the day was about 800 contracts on the S&P E-mini futures, that's $72 million of notional inventory for a typical firm on a single security. I would say that certainly constitutes a meaningful position.

"Do people know to knock on the door of an HFT firm to do business in a particular stock?"

They don't need to. Electronic markets and Reg NMS assure clients that they are receiving they are buying or selling at the best quote in the country on any stock at any time between 9:30 and 16:00. A system where people need to go door to door to find a counterpart is fraught with much higher matching costs, much less transparency and worse prices than one where anonymous traders are automatically matched at the best quote on a fungible asset.

"The seller did not advertise that some of the offered stocks would be delivered at a different price."

There is no single seller, the issue that Katsuyama had was that he was executing an order with seller A and expecting seller B not to change the price in the future based on publicly available information. Are you saying that the sellers of the stock don't have the right to change their price based on publicly available information? If market makers didn't have the right to adjust their quotes based on public trades they would have to charge much higher prices. Imagine you represent a large group of hotel room buyers, you go to one hotel and book all their rooms for the weekend. Then you call up the next hotel and find that their rates have changed since before the original booking. Did they "cheat" you or were they just responding to supply and demand? I would say that you negotiated the rooms in a stupid way, you should have either tried to book them simultaneously before the other one found out, went to a hotel that could fulfill your whole party, or patiently booked your rooms over time to not make such a splash.

"Markets were liquid for a very long time before there was HFT."

In 1995 the S&P 500 traded 346 million shares a day on average. In 2013 that number was 3.35 billion shares a day. This is a simple but reliable measure of liquidity, volume is a measure of the total capacity of the market. Of course as liquidity becomes cheaper, transaction costs are lower and the number of transactions tends to rise. By this simple metric the market is nearly ten times as liquid as it was before the advent of electronic trading. You could look at similar metrics like bid-ask spread or the average market impact of a trade, and they'd tell the same story. Liquidity has substantially improved. What you're saying is equivalent to someone saying something like "computers pretty much do the same thing they did 20 years ago, I don't believe that semiconductors have actually improved."

"If you are so certain of the benefits of HFT trading, and would like us to evaluate the information that you have provided on its merits, then identify yourself or your company. Stand behind your product."

You wouldn't know my firm even if I told you. We're a small firm with no public facing capacity or product per se. We have no clients or investors, we trade proprietary money that trade on electronic markets as anonymous participants. I used to work for a much larger firm, that you probably would know as its one of a handful of names that are gigantic in HFT. But that's irrelevant, I have no interest in betraying my anonymity, nor should you feel I have to. I'm not telling you to trust anything I'm saying on personal reputation, everything I'm claiming should either be a consequence of basic financial economics, cited, or easily verifiable on Google. If you feel otherwise please point out specifically and I'll try my best to substantiate.


I think @Mort Dubois' "identify yourself or your company" demand was silly and uncalled for. I, and I think many others on here, do appreciate the interesting and illuminating comments on HFT you've had to offer.

@Mort Dubois: Anonymous posting in blog comments is hardly novel. Further, even if he were to identify who and where he's from do you have some magic way to verify and validate his claim? So why not discuss the substance of the matter?

Wasn't trying to be offensive, and my apologies if it was received that way. I think that it's useful for anyone who provides information that we are relying on to understand a complex problem to identify themselves so that we can take into account any incentives they might have to shape their narrative. HFT has done a wonderful job of presenting a certain set of facts, but we aren't in a position to judge whether he has a reason to omit or de-emphasize other parts of the story unless we know who he is.

Ironically, "Mort Dubois" is a pen name I have used for many years. When I started, there was no reason to reveal my real name, as nobody would have heard of me. Subsequent to Mort's creation, I ended up with a regular gig writing for the New York Times - my real name is Paul Downs. So if I took HFT at face value, the very existence of the book, and its prominent placement on 60 Minutes and the Times, is inexplicable. I'm not so stupid as to believe that the media doesn't get played like a piano now and then, and this may be one of those occasions. On the other hand, reputable people look at the same set of facts that HFT has presented and feel that there is a problem. What to make of this disparity? We know who stands on one side of the argument, and what they stand to gain (sell books and newspapers.) Who stands on the other?

Even if you don't choose to reveal yourself, HFT, I do have to congratulate you on a very interesting and illuminating series of comments. Thank you for the effort you put into educating civilians like me.


First I'd say that while people like myself certainly have an interest in defending HFT, the common detractors you hear quoted in the media almost always have a (often hidden) interest in vilifying it. To a tee every serious anti-HFT advocate I've seen is selling an overpriced and unnecessary product based on grossly exaggerating any reasonable harm that HFT does to small investors. Or they're lobbying on behalf on an in interest group that would financially benefit from additional regulations. While you may be unconvinced of my case that HFT is a net positive, I would certainly hope that you realize that any impact HFT has on a long-term buy and hold individual investor comes out to a rounding error. That I believe is indisputable, and if you do I would challenge you to present any serious estimate that tries to show otherwise. And if you don't dispute it, then certainly you must admit that Michael Lewis is scaremongering ordinary investors to sell books, much to the detriment of their financial health, as buy-and-hold equities is by far the best strategy available to ordinary long-term investors.

Second, I'd point you the opinions of Cliff Asness, both in this thread and publicly available in many other places. Asness is highly informed on the issue as a world renowned quant running one with much experience who runs one of the most successful hedge funds in the world. Asness is also completely unbiased on this issue, as his firm AQR does not engage in HFT and hence has no financial position in it besides as a consumer of its liquidity providing services. Yet Asness tends to have a very positive perception of the activity, surely that must also tell you something, I'd argue more so than Michael Lewis, who has much less expertise in this domain. Along those lines, as far as I know you will not find any quants of significant renowned (most of whom have no direct interest in HFT) who have the type of negative views so common in the media. That certainly tells you something.

Even if you don't believe HFT Trader, please note that the HFT critics don't provide any plausible mechanism by which this hurts small investors. On the other hand it does seem quite plausible that competition among algorithmic liquidity providers has lowered transaction costs. At a minimum anyone can see that the bid-ask spreads have gone way down.

I'm not an expert like HFT trader and I'm not even in the industry, but I've traded stocks, and in my experience the whole process seems transparent and fair and far, far cheaper than it's ever been for a small investor.

How the system could be improved is open for debate, but Lewis is scare-mongering to sell books.

One HFT trading technique I have heard of is illustrated by the following example.

There is are limit orders to sell 1000 shares at $10.00 and 2000 shares at $10.01.
Not able to see the order book, you place a market order by buy 1000 shares.
The HFT trader, sees the order book and your order before it "hits" the market and scoops up the 1000 shares @10.00 and sells you the 1000 shares @10.0075. (he could place the limit order @1.0075 to sell.)

Why is this not frontrunning? Or is this an inaccurate description of the strategy?

What you are describing is just plain impossible. Nobody sees your order until it's completely matched against any marketable liquidity on the limit order book. Period.

I'm not satisfied yet that I understand this as well as I want to but thanks to everybody, especially HFT Trader, for a remarkably lucid and civil discussion on a controversial topic here.


@HFT Trader: Nice posts! Thanks for your comments.

@HFT Trader

As an aside, What's your opinion on strategies like quote stuffing? Or is that passe / banned now?

When I read it described it sounded very similar to a denial of service attack.

Re: Quote Stuffing. 1) It absolutely is a purely disruptive and non-productive strategy that worsens market quality. Basically a DoS attack. 2) It is a highly niche strategy practiced only by a handful of small firms that aren't good enough to make money in normal HFT. It constitutes an exceedingly small proportion of overall HFT activity. The major firms would never risk participating in it, as the risk of losing their right to access the exchanges overwhelms the tiny reward of an activity that would only slightly add to their profit.

3) The exchanges to placate their customers have gone to great lengths to prevent the activity. This includes penalizing firms who quote too often or have too low fill rates, explicitly cutting off the connection of firms that exceed a threshold message rate, and upgrading their infrastructure to handle without disruption any activity below these thresholds. 4) It was mostly an artifact of previous years when computers and infrastructure weren't faster enough to process quote activity that could possibly be at the upper end of economic. Now that exchange infrastructure is much faster it is virtually impossible to do at any major exchange. 5) All this combined means that currently well over 99.9% of trading activity in American equity and futures markets is un-manipulated from a latency perspective.

The quantitative findings of the paper are exactly as I described. You can check the figures and math yourself. Another way to rephrase this is directly from the paper: "[R]oughly 74.7% of all US listed equities experience at least one event during the 2010 trading year... The majority (72%) of events lasts less than one minute." This seems like a lot: most firms are quote stuffed one or two times a year, which roughly affects them for one or two minutes. But think about it that means the typical stock is quote stuffed for about four minutes of trading... out of 98 thousand trading minutes that occur in American trading year.

Four minutes out of ninety eight thousand, the authors own findings, does not exactly merit the term pervasive in my opinion. My sincere guess is that the authors chose that word because it made the paper more provocative and hence publishable. Which is why you should always read the actual quantitative results, and not just the subjective abstract. Regardless of whether pervasive is the right word or not, against 99.99%+ of trading is unaffected by quote stuffing.

As a laymen, this is exactly what I comprehended from hearing a Michael Lewis blurb on NPR. I see the writing on the wall: industry interests and the anti-wall street crowd joining together to regulate HFT to death.

HFT Trader: Thanks for clarifying this issue. I finally read the Nytimes summarizes M. Lewis' book. It sounds like in reality people "infer" an order, knowing that large orders end up split up among exchanges and some pieces of the order hit very slightly earlier.

This is quite a bit different than a pure front-run, and just a high tech variation of what many firms have done in the past when they see evidence of a large player.

I did work at a hedge fund that wanted to unload a significant position in a bond issue. They had several traders simultaneously call the sales rep at different brokers to do the trade (referred to as a "drive by shooting"). The spread quickly blew out, leaving the brokers holding the bag. I'm sure later quotes to the firm had additional spreads built in.

Perhaps a benefit of HFT is the inability of a firm to do such a "drive by shooting" may enable others to offer cheaper execution.

1) Supply curves slope upward. So a 'market price' of 47.64 is really for a small amount at the top of the book, which is usually scattered among several exchanges. If you want to buy 10,000 shares at 47.64, and the offer amount is 1000 you will probably have to pay more than 47.64. If you want to buy the entire 1000, if you send your buy order to one exchange you should not expect to get your 1000 filled; Katsuyama's incredulity on this point is rather naive. Further, his solution for hitting all exchanges simultaneously is well-known, so I hope he has some deeper understanding of these markets or he is out of his league (sweeping the market at current prices is feasible as he noted but usually you have many more shares to do, so you have to weigh this signal vs. simply not getting filled this second at the current price and anticipating the resupply; his solution is rarely part of an optimal tactic).

2) People think others should act strategically when they transact with them, but there are informed traders and uninformed traders in these exchanges. Uninformed traders are placing amounts that are basically uncorrelated over time, whereas informed traders are front running something (themselves, a breaking story, a clever interpretation of the news); that is, informed traders are trying to get in front of trades they correctly anticipate going the same way. A market maker has to respond strategically to keep from trading too much, at too stale a price, with informed traders.

Both facts of markets, of life. Competition is the only way to mitigate their effect on consumer surplus, and all-in costs (commission + spread + trade impact) for equity buyers has never been lower in the US

HFT, I think Joe was referring to Michael Lewis's 60 Minutes examples where RBC sent limit orders to multiple exchanges but routinely got to BATS first due to less latency there. The HFT's were quick enough to see the trade on BATS and act on the information on the other exchanges before RBC's order reached them.

@Gordon B,

Ah, I see. Let me make a few points then.

1) This situation only applies to the largest traders. In order for this to be an issue your order has to exceed the size of the touch on one of the major exchanges. At a bare minimum this constitutes 100 shares (the round-lot), but for most liquid stocks at most times this is in the thousands if not tens of thousands of shares. If your order is below this size then inter-market order processing does not apply to you. The only traders who are regularly trading at this size are large and impatient traders, i.e. highly sophisticated market participants. This doesn't effect retail at all, in fact most retail order flow isn't even crossed on the lit markets, most is internalized at the national best price (or better) by firms like GETCO and Citadel.

2) Every exchange has an order type where you tell it not to route to any other market. That is you can send your order to BATS, and tell it not to pass on to other exchanges if you exceed the touch. Anyone can employ this order type and completely eliminate the problem. Of course they restrict their available liquidity to a single exchange, so if you want to execute very large size you may determine its worth the risk of routing to multiple exchanges. That's your prerogative either way, but if you want to realize the benefit of multiple exchange liquidity it's your responsibility to do so in an intelligent way, why should the market bend over backwards, ignoring publicly published trades, just to accommodate a few traders' voracious appetite for liquidity. (Remember the only people realistically needing this much liquidity are large and sophisticated traders).

3) Even in this case Joe's example does not hold. Let's say Exchange X has 15,000 shares at a $10 and Exchange Y has 12,000 shares at $10. I send a market order for 20,000 shares which matches at X then routes the remaining 5,000 to Exchange Y. All that's visible on the data feed is that a trade for 15,000 occurred at X. Nobody knows that this is a market order (versus a limit order), nobody knows how much quantity (if any) is remaining in the order, nobody knows whether the order is Exchange X only or not, etc. All we see is that a large trade happened, we don't know anything else about the order behind the trade. In Joe's example the HFT would sweep the remaining 12,000 shares at Exchange Y and place a limit for 12,000 shares at $10.01 hoping to earn an instant profit. But if it's a limit order for $10 instead of a market, or it has less than 12,000 remaining in the order said HFT would end up with an unhedged long position that he needs to get out. Furthermore such a position would be subjected to adverse selection and he's expect to lose money on the inventory.

@HFT Trader:

Is what you describe related to the fact that most HFTs can internally calculate & replicate the books much faster than the publicly available quote?

Would it be an option to either (a) Speed up the calculation of the NBBO from the SIP? or if not then (b) Delay the direct feeds (by tiny amounts) so that the internal books aren't systematically faster than the public books.

In some sense, yes, some edge of HFT is derived from direct (and faster) exchange feeds. Some of this is an artifact of the fact that we have multiple competing exchanges, and it's impossible to get them to perfectly sync due to the laws of physics. I'd still say the benefit of competing exchanges outweigh the drawbacks. Like I said I'm in futures space, and there every instrument can only be traded on a single exchange due to the centrally cleared nature of futures. Overall future exchanges charge much higher fees and have lower quality infrastructure, so competition is beneficial. If you actually look through the feeds the individual exchanges only show different prices than the SIP NBBO <1% of the time. So the synchronization only is a minor point for ordinary investors, and tends to only be a big deal for traders who are explicitly trying to trade around these events.

Speeding up NBBO to the extent that seems to me an unqualified improvement. But NBBO will always be slower than direct exchange feeds because it has to consolidate each feed then send it out, so it involves several hops instead of point to point communication. You could try to delay the direct feeds to sync with NBBO, but it's easier said then done. There's technical challenges to it, since with packet switched networks you can only control when you send your messages, not when they arrive. It would be very difficult to exactly sync market participants receiving the exchange feeds at the same time as NBBO.

One slam dunk reform you could make is to allow all investors to use ISO orders. These order allow market participants to trade at prices worse than displayed NBBO. Currently only broker dealers can use this type of order. So it forces normal market participants to use stale NBBO prices even if they have the direct exchange feed and know the prices are stale. Removing this requirement would go a long way to making the direct exchange feeds more "public."

Why would NBBO always be slower? Aren't the HFT's who build internal replicates of the books essentially doing the same thing? And mighty fast. The HFT's must have to consolidate & compare these feeds too if they do indeed find global arbitrage? Isn't it more a question of motivation & incentives?

Agree about packet switched networks but I'm ok with stochastic delays but the status quo seems a systematic delay with the SIP feed being stale predictably & often.

HFT's are consolidating the individual feeds but they're doing it by receiving each individual feed and consolidating at the point of reception. The SIP feed has to broadcast the consolidated feed at the point of transmission, which means that it needs to receive the individual feeds then re-transmit adding another hop. Imagine if Alice and Bob send their messages to Charlie, who combines them then relays it to Danielle. Using Charlie as an intermediate will always involve a greater distance travelled than simply going point-to-point with Alice and Bob directly.

You could add stochastic delay to exchange feeds so that the expected time is the same as SIP, but that doesn't eliminate the information advantage of direct exchange feeds. Half the time the direct exchange feed will arrive faster than SIP, and with four major exchanges, 93.75% of the time one exchange feed will be fresher than SIP. So unless you explicitly cripple exchange feeds to be slower than SIP with near certainty they'll still add significant information edge.

Again though these are incredibly minor issues. The below paper examined typical US equity feeds and found that NBBO dislocations typically only occupy five milliseconds or so per second of trading. If you're an ordinary investor who's not timing his orders down to the sub-second level, then NBBO only affects you 0.5% of the time, and its overall cost to you is basically a rounding error. The NBBO issue is only serious for heavy algorithmic traders who are carefully timing their trades at the millisecond level, virtually all of these players are already receiving direct exchange feeds anyway.

@HFT Trader

What if we no longer allow Alice to talk directly to Bob (for a fat fee, of course) but only via Charlie? I'd love to see a quantitative estimate of how much liquidity-loss / accuracy-loss we'd expect from those few milliseconds of added latency because of an extra packet hop. Know any studies?

You've referred to these as "incredibly minor issues" or something equivalent. Well, first, if they are so minor how does one explain the incredible amounts of money spent on fiber-wars, co-location spaces, and TCP offload cards etc. Alternatively, if they are such a minor part of HFT strategy the companies may not object too much to any slight changes to protocol?

I always feel the devil's in the technical details: e.g. what about the enhanced feeds exchanges are rumored to sell to the HFTs for a price? Is that true? Also what about the use Flash Trades to scoop up orders by HFTs before they are rerouted? Are these practices liquidity maximizing too?

Finally, a naive question, perhaps nitpicking: If the SIP NBBO is stale so often, how is there any practical enforcement of the Reg NMS rule that a quote be re-routed to the best exchange available at that tick? Is there an audit trail kept & even if so, with a stale SIP book what good is it?

When I said these are "minor" issues, what I meant are they are minor for ordinary investors. To HFT investors these are quite serious issues, but they're not serious because they allow HFT to gain a leg up on ordinary investors. They're serious because they're about HFT vs HFT. Putting a strategy in hardware or laying direct fiber or getting a direct feed is about being the first player in line who gets to act on some information. Even if these things didn't exist a computer will always be faster than a human, but it's about making your computer faster than other computers. As a human, a computer being 1 millisecond faster has no effect on you because the computer was already at least 500 milliseconds faster to begin with.

Going along your suggestion of only allowing HFT to listen to the consolidated feed, sure it could work. I doubt the impact would be that large, but going along with what I said above it would have no impact on ordinary investors. It would only change the intra-HFT dynamics. They would still compete fiercely to be faster, just not with direct feeds. In futures space there's only a single feed and its just as fierce if not more so. The direct feeds are a revenue source for the exchanges, so there'd be less revenue for the exchanges and more going into faster hardware to process the centralized feed. That might be a good thing, or it might be a bad thing because some of the "wasteful competition" that goes into positional advantages probably lowers exchange fees (which benefit ordinary investors) by increasing exchange revenue for their feeds.

With regards to special feeds or secret feeds, no they don't exist. I don't know how "prove" it to you, but I've worked at one of the largest and most sophisticated firms and there were no secret feeds. As for flash orders, they were discontinued at least three years ago. They're a quite complex topic that deserved a nuanced discussion and were unfairly vilified. But I think it's rather moot because they're no longer relevant.

It's a good question you're asking about stale NBBO. The complexity of the issue is one of the reasons there's such a huge number of order types in equity space (compared to futures space, where there's basically just limit, market, and IOC). Reg NMS is overall a very good rule that's positively benefitted 99% of investors. Again referencing my previously cited paper the typical stock is only out of sync less than 0.5% of the time. Since ordinary investors are as likely to send an order as any other time, and as likely to buy of sell, it only affects about 1 in 400 trades. In contrast before Reg NMS the brokers played a lot of games with customer orders where they would sit on them and wait to see if the price moved or not (in effect giving themselves a free option at the expense of the customer).

The exchanges are required to honor SIP NBBO even if they know it might be stale. So it can be somewhat absurd, but there's not really any other way to enforce honesty and competition. To my knowledge they do keep meticulous records. Like I said my only major criticism is that accredited investors who know what they're doing, and not just broker dealers, should be able to use ISO orders to trade at worst prices than NBBO if they know it's stale.

HFT Trader, thanks for taking the time to post at length on this thread.

If you'll endulge me, I have a question regarding fills. I very often (maybe majority of cases) get orders filled at fractions of pennies variance from an actual limit order. For example: buy order for AFL filled at price of 62.3299. I usually use limits when buying/selling so I'm guessing my limit was 62.33. Can you help explain what's happening behind the scenes here?

I'm picturing arbitrage where somebody buys at 62.32 a split second before my order fills, and then sells it to me at 62.3299 to step in front of all the other orders willing to sell at 62.33. (At all my brokerages I can only input full penny order prices, so I'm not even sure how the fractional pennies come about). Ultimately - I'm wondering is that scenario likely going on? Do the HFT algos make guesses all day long scraping pennies this way (if that is what is going on)?

I trade mostly with Scottrade, but also trade with Wells Fargo and Tradeking if that matters. I've come to often expect fills w/ fractional pennies at all of them.

Thanks for clarifying earlier also that HFT can't "see" my order coming before it gets there. To an outsider it often seems like that happens - the bid/ask will move adversely almost synchronously as I enter an order sometimes.

Typically what happens is that most retail brokerages sell their order flow to internalization pools run by large HFT firms. Internalization pools are a type of dark pool where a HFT firm is directly trading against order flow without passing it to the exchanges. Since retail traders are highly uninformed their trades rarely move the price of the stocks they're trading. Most market makers would much rather trade directly with retail order flow, then the general order flow found on exchanges, since there's much less risk and adverse selection to it. Reg NMS says that they regardless they must meet or beat the best price available in any exchange at the time of the order. Most pools will slightly improve the price by $0.0001 to make sure they stay in compliance.

What is happening is you place a limit buy order for $62.33, and Scottrade sells your order flow to an internalization pool. The national best offer is $62.33 so the internalization pool slightly improves the price to $62.3299. The internalization pool has earned the spread off a basically uninformed trader so it's much happier than trying to sell on the exchanges at $62.33 where it's subjected to being picked off by sophisticated traders. Scottrade is happy because it got paid something for the order flow, and you're happy because you got filled at slightly better than the best national price and pay a lower commission because Scottrade is make revenue on the sale of your order.

The "losers" in this scenario are the market makers who are quoting on the exchanges, since the internalization pool captured some of the uninformed flow they are now subjected to more informed order flow. Also the informed traders will pay higher trading costs because no one wants to internalize their trades so they must trade on the exchange where liquidity will be slightly more expensive because of less counterbalancing uninformed flow.

A refute to HFT claims from the guys at Nanex -

I don't know what "etc." is, but like I said quote stuffing is already banned by every major exchange, and it's a good rule.

The exchanges explicitly prohibit an excess rate of messaging. How does that not constitute banned?

Some market participants will try to engage in it and be successful. That doesn't mean it's not banned, any more than the fact that some people smoke weed means that marijuana is legal.

And to reiterate what do you mean by "etc."

On a lighter note, I loved this description of HFT:

Customer: How much are those?
Merchant: A buck fifty.
Customer: I'll take some.
Merchant: They're a buck fifty-one.
Customer: Um, you said a buck fifty.
Merchant: That was before I knew you wanted some.
Customer: You can't do that!
Merchant: It's my shop.
Customer: But I need to buy a hundred!
Merchant: A hundred? Then it's buck fifty-two.
Customer: You're ripping me o ff.
Merchant: Supply and demand, pal. You want 'em or not?

On a lighter note, I loved this description of HFT:

Customer: How much are those?
Merchant: A buck fifty.
Customer: I'll take some.
Merchant: They're a buck fifty-one.
Customer: Um, you said a buck fifty.
Merchant: That was before I knew you wanted some.
Customer: You can't do that!
Merchant: It's my shop.
Customer: But I need to buy a hundred!
Merchant: A hundred? Then it's buck fifty-two.
Customer: You're ripping me o ff.
Merchant: Supply and demand, pal. You want 'em or not?

Comments for this post are closed