What are the social costs of high-frequency trading?

I’ve yet to see a good argument that they are high.  HFT is taken to mean many things, but let’s (for now) focus on high-speed arbitrage and near-arbitrage.

Let’s say the market for coconuts in Thailand reacts somewhat slowly, and the market for coconut derivatives in Singapore allows for quicker trading.  A storm comes to Thailand, the two coconut prices split, and a number of traders rush in to take advantage of the price discrepancy.  (Of course since the Thai market is slow and less liquid, this won’t be perfect arbitrage.)

If ten traders have more or less the same speed (and quality) of trading technology, the returns to rushing would appear to be pretty small.  At most, the $$ invested in speed will rise to equal the size of the available p x q discrepancy.  That’s basically the same result you get with slower trading technologies.  Call it waste, or not, but I don’t see that any new problem has arisen here.  There is some waste, bounded by the p x q discrepancy, whether people compete over speed at higher speeds or lower speeds.

If one trader has dominant speed, that seems to also limit the costs of running after the arbitrage profits.  Rent exhaustion will be far from complete.

Alternatively, imagine a leapfrog model.  The quickest firm gets to be clear leader for a year, but by the time that year is up they are leapfrogged by a new and speedier technology, and then there is a new leader.  It still seems to me that the investments in the new speed technologies are bounded by the p x q discrepancies, as they were in slower times.

Keep in mind, if HFT yields profits, there are also incentives to improve the trading technologies in the slower of the two coconut markets.  Those incentives will limit the profits from HFT and thus the resources invested in HFT.

I understand full well that this discussion considers only a few relevant factors.  Nonetheless I don’t see that the critics are imposing even this much structure on the problem.  I don’t see why “at higher speed” makes the rent exhaustion problem from price arbitrage more costly in social terms.  I don’t see a good theoretical or empirical argument on the table, much less a verified argument.

You also might think that more volatile intra-day asset prices are a cost of HFT.  Hold off on that for now, I’ll consider it in another post.


What if HFT exaggerates market imperfections that are structural (maybe because of limits to arbitrage as Shleifer described)? I'm not making the case, by the way, just wondering.

When I discuss HFT with friends (none with a background in finance or economics), the discussion of costs mostly centers on what will happen when something goes really wrong and due to large trade volume and high frequencies that cannot be fixed before it causes damage an order of magnitude larger than "The Knightmare", i.e. the costs of the coming bailout.

Please ignore "that" between "frequencies" and "cannot"

Some HFT algos use fake bids. We don't need junk information, we need real information, otherwise they are spewing pollution.

Placing fake liquidity to manipulate the market doesn't require automation, just anonymous electronic trading. The actual examples of market manipulation I'm aware of all involve humans trying to manipulate algorithms, not the other way around.

HFT trading makes this type of manipulation behavior harder, not easier. Computers that can sift through giant datasets have a much easier time building models about the characteristics of manipulative orders. A machine that can analyze thousands of variables in determining this is going to do a lot better than a human eyeballing a limit order book.

Hence HFT algos are much more likely to identify fradulent liquidity and will confidently cross it, incurring a large cost to the gamer.

Is that why volatility has gone through the roof while liquidity has collapsed? The good and honorable HFT have eliminated all the gamers?

Volatility is high because of a little something called the global financial crisis. Blaming the people who make the markets is just another twist on the old meme of blaming the speculators during a famine.

The 'father' of HFT was interviewed on a Planet Money segment a couple of weeks ago and said that we should slow down: http://www.npr.org/blogs/money/2012/08/21/159373388/episode-396-a-father-of-high-speed-trading-thinks-we-should-slow-down

Go figure: he's good at talking his book, like every other financial institution and company in the world.

Now that he's made $5.4B from just this kind of automation, he's "in favor of regulation." Color me surprised.

Alex linked that yesterday, this guy claims he's a pioneer of electronic trading, not HFT. And that he's wrong.

As Wonks Anonymous stated, Petterfy is not a father of HFT at all. He has stated in other public forums that HFT should be forced to slow down, while he, as a market maker, should not be forced to slow down. How hypocritical is that!

"If ten traders have more or less the same speed (and quality) of trading technology, the returns to rushing would appear to be pretty small. At most, the $$ invested in speed will rise to equal the size of the available p x q discrepancy. That’s basically the same result you get with slower trading technologies. Call it waste, or not, but I don’t see that any new problem has arisen here. There is some waste, bounded by the p x q discrepancy, whether people compete over speed at higher speeds or lower speeds."

That is certainly true, however if we imagine that the gain to society of closing a particular discrepancy, d, with speed, v, is declining, such that we can write that d*Sqrt[v], and that the cost of increasing said speed is convex, then from a societal perspective we'll be optimizing: d*Sqrt[v]-v^2.

If however from a private perspective, the gain is constant, so that the firms are optimizing d*v-v^2, then there will be waste from a societal perspective.

I think that those who oppose HFT are asserting that the marginal gains to society are smaller than the marginal private gains and so therefore further increases in speed beyond a certain point are pure waste. Even if that 'waste' is disciplined by p*q.

Their solution is to put a maximum on v, equal to the societal optimum.

Note that the gain here is expressed per unit of time.

A lot of good brains are wasted on a zero-sum game. Oh, and quite a chunk of good money spent on needless infrastructure

This is by far the most convincing argument against HFT. I'd like to see pro-HFT person's response to this.

Responding to this is not that hard: besides the question of whether or not it is a zero-sum game (I think it is not), who are you to define what is and is not a "waste" for a good brain?

Well, the brain waste is predicated on it being a zero sum game.

And, I'm another brain! We're all our own personal arbiters of personal truth and beauty.

You need not believe it is a total waste to believe there are higher value uses of very smart people's time than playing zero sum games against other very smart people. I think most people who engage in HFT have high opportunity costs of time.

This is predicated upon my view that the average person playing the HFT game at a high level is at least as smart as the average person in any other profession (including doctors, lawyers & professors).

It's like professional sports. In order for me to win, you have to lose. Doctors don't need other doctors to lose in order to be successful themselves.

Don't have a strong opinion on HFT (not my area), but, even if it is zero sum or indeed negative to society as a whole, there's no stable equilibrium to discourage it - trading faster than others is always in *my* interest, within the bounds stated in the article at least, so I'm going to do it.

It's not zero sum. Spreads are tighter because of HFTs. Investors pay lower costs to useless intermediaries because of HFT.

HFT is not a zero sum game. It's not possible for it to be a zero sum game in the real world. It's a business that sells a product to a customer. The product is prices that are more accurate, given what's knowable at the time, than they would be if HFT didn't exist. The customer is the market, and HFT is paid indirectly for what it sells, by transacting in the market. The market, a self-organizing system, only pays for what it needs to survive, and pays nothing for valueless things. The market is willing to pay for accurate prices, it's one of the lifebloods of its existence. HFT is a highly competitive business, and behaves like one. This means that the value it extracts as payment for producing what it sells, is far less than the utility it delivers to the market. This is the common reality of all competitive businesses. They always get paid only a small fraction of the value they create for their customers. This is a simple consequence of competition, which drives the cost of production to the limits possible given the state of current technology. The benefits of prices that are fairer for all participants, summed over all transactions among net buyers and sellers, dwarfs the profits of the HFT business. It's a very big mistake to think that it is a zero sum game. The most vociferous critics of HFT are businesses that previously were able to extract monopoly rents from retail customers, usually by hiding large orders they were moving through the market ("best execution" firms are a good example of this). By hiding the size of their orders, they were able to dupe the other side of the transaction into paying more to buy, or receiving less to sell, than they would have if they had known about the size moving through the market. Now, because of HFT, these same firms cannot hide as much information, and the other side of the transaction gets a fairer price because HFT has discovered some statistical information about the large order and has adjusted prices accordingly.

The better question is not Why do smart people continue to be attracted to Wall Street like moths to the flame, but rather, Why are smart people surprised by that attraction?

Twenty plus years ago Michael Lewis lamented:

"seven thousand five hundred college and business school students [applied] for the six jobs on the trading floor of Solomon Brothers (compared to six thousand for 127 jobs in 1985)" ("The Money Culture," 1991, p. 59.)

Lewis' thesis then was that "the age-old taboo of handling money was shoved offstage by the sheer force of events in the financial world, clearing the way for a new money culture" (p. xiii.).

But with the benefit of years we can see that there is little that is "new" about the denunciation of Wall Street. Reaction to the story of Jesus overturning the money-changers' tables has never been unpopular.

And Socrates' view, as quoted in The Republic (Bk. II, 369b-371d), that since no able-bodied man would agree "to sit idly in the marketplace," traders would need to be culled from "those whose bodies are weakest and who aren't fit to do any other work" is a put-down still widely held.

So, the question is not why does trading continue to attract bright young people, but rather, why is the popularity of stories that denounce trading never out of style? Why is Wall Street-related employment always considered in polite circles to be contemptible?

This is a better argument against sports than HFT.

Agreed...add in casinos and bingo halls and concerts arenas and the Olympics and...

Very few good brains are wasted in sports...

Konstantin - absolutely right. It's socially not useful to spend that much money/brains on beating the market. (Paul Samuelson made this point decades ago - something like "we should all stop looking for needles in a giant haystack").

Of course it's zero sum, any purported benefits from liquidity have been shown to be a mirage. (e.g. this http://www.bankofengland.co.uk/publications/Pages/news/2011/068.aspx)

To answer Tyler's question: there may not be much social cost from HFT, but there's zero social benefit, with resources diverted to unproductive means.

Society determines what is a "waste" all the time, rightly so. We encourage children to go into education in particular subjects. We regulate gambling. We put limits of addictive substances. In this case, I don't think people know it's a waste of resources.

The comparison with sports is instructive (even though I think Alexei meant it in a negative way). We get entertainment from watching sports, even though it's a zero-sum game. We enjoy the thrill and uncertainty. If anyone gets entertainment from HFT, it's the market participants, not the end-investor (pensioners?! Who is the ultimate investor?).

Andrew: Banning or discouraging an activity because it is harmful is fine, and by all means you should advocate for it. And there is an ongoing discussion about the benefits and harms of HFT. In which direction Haldane's paper is a particularly silly addition: he speaks of rising correlations and volatilities in the last few years without nodding once in the general direction of what has been dubbed the Great Recession, with all its panic driven sell-offs, euphoric buy-ins, and widespread risk-on / risk-off behavior. And he cherry-picks a start date of 2005 to mislead the reader into thinking the effects he talks about extend beyond the recession, even though all his charts show the paradigm shift happened in 2007.

However, banning an activity, even in the absence of any evidence of harm, just because in your opinion its practitioners may be better employed elsewhere is even sillier.

One can also argue, likewise, that being an economist, say (to pick name completely at random) Samuelson, is a waste of a decent brain that could, with some effort, become a halfway decent physicist or engineer or mathematician.

Sports is entertainment, not zero sum. The NFL makes more meny every year even though only one team wins the Super Bowl. And fans get lots of value as each generation of athletes raises the performance bar.

Well certain expenditures in sports are zero sum. For example, every team now has vast computerized databases of opposing teams plays. They spend a ton of time and money on it, but the end result is mostly a wash. Fans don't watch and go "Ooh I love to see that Viking film database in action".

Or more specific to the world of sports, the wildcat formation. When Miami first started it, it was unique and exciting, and provided value to the fans. Now that 1/2 the league is doing it, it's no longer fun. It's just a tax on the time of defensive coordinators and personnel, who need to spend a lot of time putting in a specific package to counter something that may be used only a handful of times a game. I'm not in favor of super specific rule changes of the sort it would take to eliminate the wildcat, however if I could wave a wand and eliminate it I don't think that many people would miss it. All that would happen is that a bunch of defensive personnel would get more sleep during the season.

So the NFL as a whole generates a lot of value to society. However many technologies and tactics used by the teams can be considered zero sum.

This is one of my concerns as well. It's the opportunity costs. We're spending a lot of time, money, brainpower, and resources on HFT. What else could have been done with those things?

TGS! It is not obvious what the opportunity costs (at least for brainpower) are...

Just to play devil's advocate: engaging in nefarious, subversive activities?

(Though I agree, this race to slice milliseconds from transactions is a huge time-, resource-, and brainpower- suck of an endeavor with no social value).

TGS doesn't have to be our future. Send the comp sci guys to work on automated cars, or to bring about the singularity. Recast the physics guys as material scientists and have them work on high temperature superconductors.

A lot of good brains are wasted trying to figure out how to get Facebook ads to have a slightly click rate. A lot of good brains are wasted trying to figure out the placement of cereal on the shelf in the grocery store. A lot of good brains are wasted trying to figure out how to maximize the amount airlines can charge on fares based on the time of day. A lot of good brains are wasted on post-modern deconstructionist literary criticism.

At the point that you're proposing regulating something not for any specific market failure, but simply because you don't see the use in it, you're strayed pretty far away from free-markets into the realm of central planning. A place that the vast majority of economists and ordinary Americans would not want to be.

This is the best pro-HFT argument I've seen, though I dispute that there have not been any specific market failures. The 1987 crash can be attributed in part to that era's equivalent of HFT. Programmed selling. If 1987 had never happened HFT would not get as much scrutiny.

One could make a serious argument that the need for ever faster HFT infrastructure drove Sun's profitability, allowing them to create, and then freely distribute Java.
Sun's warchest and the outstanding ideas contained within Java brought Microsoft to heel* and have brought us to the point of the $100 smart phone.
If you really want to stretch, we can throw in Sun's purchase of MySQL and Open Office, both of which exerted price pressure on Microsoft.

I don't suppose there are any gains to society there?

*The .NET framework is Microsoft's improved re-implementation of Java. A significant portion of Sun's revenue in the post-crash time period was the licensing fees
that Microsoft was paying them for the patents/IP that said implementation required.
An additional condition of the agreement was that Microsoft had to seriously work with Sun to ensure their stuff was interoperable with Java. Before "Metro" was the
new user-interface for Windows 8, it was the RI web services stack from Sun that had an interop guarantee with the .NET stack.

"HFT: it gave us Java"? That's a defense of HFT?

And the flash crash 2 years ago too? Wasn't that HFT?

Whoops that wasn't supposed to go here. This comment section is weird.

Wikipedia says that HFT started in 1999 when electronic exchanges were authorized. Maybe that's true, maybe its not. It's pretty tough to argue that HFT was driving a lot of Sun sales before 2002, when Java 1.4 was released.

Isn't that true of whole of finance and banking? Not just HFT.

Also, the needless infrastructure sometimes turns out to be game changing a decade later. In the mid-nineties when the banks started creating creaky looking websites, who would have thunk that a decade later (around 2005 say) you would be able to do large amount of routine financial transactions (deposit of paycheck, payment of bills on time) online, on time, automatically. Who knows how this round of manic infrastructure building will pay off?

The problem is not so much the answer to
"What are the consequences when the algorithms work exactly as their creators intended?"
The problem is
"What happens when the algorithms behave in unexpected ways?"
When a human trader makes a trading error that causes the market to react in an unexpected way he or she will certainly pull back. An trading algorithm will not.

This kind of argument could be made against any type of automation, for example robot drivers (a favorite topic on this site), not just automated trading. I suspect that on balance, the robots
(whether traders or drivers) will do a better job, and commit fewer errors, than their human counterparts.

That's a fair point, and clearly human traders in financial markets can and do make mistakes all the time. And your right, it could be used as a (bad) argument against automation in lots of contexts.

But I do think it's a better argument in the case of algorithmic trading. Unless the algorithm has been programmed to understand perfectly all relevant feedback, it can lead to errors of a different kind than those that humans would make.

Recently there was a case of where the interaction of two price setting algorithms caused a book on amazon to be priced at over $23 million. (Link attached).It's hard to believe any human would have made the mistakes that led to that occurring. It's a curiosity in this case. But what if it was a financial instrument used to reference millions of legally binding derivative contracts? What if automatic margining running in the background led to a liquidity squeeze that forced financial institutions in to default? What if clever lawyers were able to argue in defence of private interests that benefitted from the price spike and the legal uncertainty dragged on for years. Now one response is to say "We can think of regulations to negate the impact of those specific potential issues." But we have a rubbish track record of trying to design fine tuned future-proof financial regulations. Better to find structural ways to address potentially large issue even at a cost of slightly less financial innovation.

The $23m was an offer not a bid and thus was completely harmless. I can offer to sell a facebook share for 1 billion or 100 billion and it will have zero effect on the markets. I can put an ad ion craigslist offering to pay 1c for a brand new Ferrari and guess what - it's not going to affect the business of Ferrari dealers anywhere. It seems knowledge of the simplest market mechanics is not a pre-requisite to criticize hft. no disrepect.

Hi Derriz, believe it or not I've worked in/around financial markets for 15 years. My point (which I think probably wasn't very clear) is that algorithms can interact with other algorithms - and with other things in the world - in surprising ways that human programmers didn't forsee. I wasn't claiming that anybody bought the book for $23 million.
I've seen close up an algorithm interacting with another algorithm, that constantly buys and sells the same contracts at the same price, simultaneously buying and selling a longer dated contract at the same price. That's not too serious - it means volumes are artificially bit higher, and it means any client money gets used up in any direct transaction costs more quickly, but ultimately it's not going to break the financial system. But the programmers of those algorithms didn't foresee that particular set of problems. So there may be other sets of specific problems they didn't foresee.

I think the main social cost of HFT is that it can reduce the investors (esp. retail investors) faith in the market.

Exchanges give preferential treatment to HFTs (e.g. co-location, ability to use more order types etc.) because HFTs lead to higher volume and higher fees for the exchange. Arguably, as businesses, Exchanges have a right to discriminate this way between customers based on pricing and they have been doing so ever before HFT. But at some point this leads to a perception of the market not being "fair". As an investor, I may accept that a higher fee paying customer can see instant prices while I get a delayed quote. However giving someone the ability to "see" my order makes me a little more queasy.

The loss of faith only increases when these black boxes fail in spectacular manner such as during the flash crash and "Knightmare".

Since we have had at least two high profile failures causing all manner of havoc in the system, shouldn't we ask for proof of a social benefit from this activity before allowing it? Of course my concern is more with algo trading of which HFT is a small subset.

The irony is that this "loss of faith in the market" seems to be more caused by alarmists kicking up a fuss about things they don't understand (and many of whom have no incentive to try to understand, because HFT is undercutting their business).

Let me put your mind at ease at least a bit -- nobody is "seeing" your orders before they hit the market. That would be completely crazy and would make it impossible to prevent front-running. The exchanges, brokers, and regulators have NO rational reason to allow something like this, and I can tell you straight out that it is not allowed on any exchange. The HFT firms' colo allows them to get the fastest possible information about trades that *have already happened*, or quotes that you *intend to publicly display anyway*. The world may be a crazy place in some ways, but we're really not living in some sort of conspiracy-laden proto skynet dystopia like some people seem to like to think.

Wrt the flash crash and Knight, I would point out that the pertinent issue there is just making sure that the retail guys are protected from using wacky execution algos and order types that could end up with them trading at bad prices in a flash crash scenario. If algo firm A trades with algo firm B at a flash crash fire sale price, that hurts nobody but the firm on the sell side of that trade. No you can't "crash the stock market" with an algorithm; it's not as if some firms trading at weird prices magically changes the fundamental value of the stocks being traded. Electronic markets are a pretty complex system and I'm not optimistic about any regulatory approach to prevent weird feedback loops, but I am pretty optimistic that there can be pretty robust approaches to making sure that nobody gets hurt by these things other than the firm running the algo. In Knight's case it seems like things went pretty much as you'd expect in a well-functioning system -- firm has crazy risk control failure, firm gets mauled. Other traders make a profit from trading with them. What exactly is so wrong with this?

I mostly agree with mrg, with one exception. Retail brokers like TD Ameritrade and E-Trade sell the right to get a first look at their order flow to market makers and prop trading firms before it is sent to an exchange or ecn. In exchange for this payment, the market maker has the right to take the trade. If they do, they usually improve the price by a small amount (which good for the customer) relative to the best exchange quotes. If they pass on a trade, it's send to the market. The potential problem I see with this kind of arrangement is that on the one hand it relies on the public market for prices, yet on the other hand, withholds orders (and so information) from the public markets. It seems possible that this practice makes the markets less efficient than if all orders were required to be sent to destinations where they were available for anyone to see and trade on. I don't know if this has been studied.

This kind of information hiding happened with large orders when the markets were largely manual: your executing broker would work your large order against other order flow or against his own inventory and try to release as little as possible onto the public markets. With the advent of automation, this has now become feasible for small orders as well. The technological enabler is electronic markets, not low-latency / HFT. And, to the extent that this is undesirable and needs to be controlled, both the reasons and the means are orthogonal to HFT.

Except now there is a barrier to entry into the official exchanges. Unless you have the means to trade at high frequency with up to date algorithms. So it will tend to drive more volume into those order flow controls. The question will become why the exchange at all?

Yeah good point. IMO the whole "payment for order flow" model is sketchy as hell (though not because any retail guys are necessarily disadvantaged; afaik that flow is all "dark" so if you don't take the trade you don't see the order? otherwise doesn't seem like something they could get away with at all) It all seems to be a relic of the dark ages where to trade you'd have to call up your broker and he could "work" your order with all his friends... Hopefully with more direct access to electronic markets that sort of thing would disappear eventually...

well said, MRG, I very much agree and couldn't have said it better. thank you.

"this “loss of faith in the market” seems to be more caused by alarmists kicking up a fuss about things they don’t understand"

mrg, is that what everyone was doing back in the fall of 2008 "kicking up a fuss"? I agree that financial products/strategies whether it be HFT, derivatives, MBS, no-doc loans, etc. have aspects (good and bad) that are not widely understood. And yet even people who "understand" these products don't always understand (or admit to themselves) just how fragile some of their assumptions might be. Many of the "alarmists" today helped pay indirectly for a bailout that market forces "should" have prevented in other financial products. I am not necessarily making a critique of HFT, but I think the views of alarmists should be considered (and the market will fix it is not good enough for the systemic risks). History does not repeat itself, but we could pick up a few lessons along the way, don't you think?

One other related comment. Here's a WSJ post that talks about early access to data for HFTs: http://blogs.wsj.com/marketbeat/2012/08/28/high-frequency-traders-flat-out-buying-data-ahead-of-you/ Strategies that rely on information asymmetries make me a bit uncomfortable, since there's an incentive to create new asymmetries and reduce transparency.

That is (unfortunately) an entirely reasonable perspective to take from an outside vantage point. It's certainly true that the alarmists are sometimes right and they shouldn't just be dismissed outright. Certainly there is a need for intelligent regulation. However as an industry "insider", I know enough to know that this particular fuss over HFT has gone way beyond mere lack of understanding to the point that *most* of what HFT critics claim is just outright factually false. (Actually the discourse does seem to have improved somewhat recently, but I'm afraid that it'll be a long time before the industry recovers from this public lynching). To trace the causal chain back as far as I can see, this unbelievably misleading hatchet job of an article http://www.nytimes.com/2009/07/24/business/24trading.html is what kicked off the whole thing. Once there was blood in the water there was really no stopping the uproar; after all why bother to fact-check anything when you can nail those damn wall street villains to a wall by repeating the same absurd claims over and over? Just look at all those juicy lines you can cite from the NY Times; must be a real legit problem right? It's been a very frustrating experience for me to see this happen, but it's certainly opened my eyes to just how much you can trust journalists to give you even an approximate view what's happening out there in the real world. I'm still not sure if this was intentionally kicked off as a cynical campaign by the old-school broker community to try to kill off a disruptive new competitor, or if the HFTs are just so tempting a target for villification (wall street AND skynet all wrapped up into one evil package!) that this was all inevitable. Sigh.

However as an industry “insider”, I know enough to know that this particular fuss over HFT has gone way beyond mere lack of understanding

Well, insiders with a lot to gain would say that wouldn't they?

To be clear, I'm not calling you a liar and don't have a strong opinion on HFT. But as Claudia notes a lot of "insiders" right up to the Fed chairman told us everything was fine up until about 2007.

So for the time being at least the "prove-it" burden rests on insiders, not sheeple alarmists.

> Well, insiders with a lot to gain would say that wouldn’t they?

Nah. The only thing I have to gain from this discussion is a little enjoyment from arguing with folks who seem intelligent and rational and open to re-evaluating prior opinions. The fact of the matter is that the regulators are quite familiar with what we're doing, and think it's pretty much kosher. I'm actually pretty impressed by the degree to which they've completely ignored the angry mob outside our door and continue to try to carefully figure out and do "the right thing" instead of trying to appease the crazies. :)

From working on smaller systems I know this: you can't take a control system that is functioning in equilibrium and predict its behavior if it gets bumped (or drifts) far enough outside that equilibrium. Finance doesn't seem to be able to clean up after itself when it goes haywire, either. What are your grounds for optimism?

Well I was specifically being optimistic about being able to protect retail users, not about keeping the whole system stable. I think on the whole, algos are not very likely to do any harm to anyone but the trader who's using them. The non-trading public seems to like to freak out about price fluctuations as if the "price" of a stock were some real physical thing, when in reality all they're usually looking at is the last price some pair of counterparties traded at (15 minutes ago). If you bought some GOOG stock and some idiot's algo flash-crashes it down to $1, it's the idiot who's hosed, not you. Nothing about the physical world has changed except that some shares have changed hands at an unreasonable price. Google is not suddenly bankrupt because somebody did a really crappy job of trading their stock. It'll be back trading at its normal price momentarily (in fact, that's part of the typical *definition* of a "flash crash") If you stick to using normal limit orders and don't use any weird execution algos yourself (or at least don't use any that you don't fully understand), all the algo-based instability in the world can't do you any harm. There are some details the exchanges need to work out around clearly erroneous policies and breaking trades, but I think over time they'll converge to the right approach.

To be a little more fanciful though, consider that this is probably the first time in human history when there's a full-blown industry centered around building fully autonomous agents that take independent action in the physical (well, at least financial) world without any possibility of real-time human intervention in their decision process. There is probably a good bit of science to be done here. Just because we don't know all the tricks to keep these agents under control doesn't mean we won't figure them out over time. Hell we'd better, because if not we're going to have much more serious problems on our hands as our gun-toting millitary killbots become more autonomous. :)

If you bought some GOOG stock and some idiot’s algo flash-crashes it down to $1, it’s the idiot who’s hosed, not you.

Isn't there the potential that if you are buying on the margin, you'll be forced to sell into the fall? Then other people's crash can hose you.

I think the real problem is also the danger that a flash crash causes either real panic or algo panic. In a volatile economic market, a false call of fire in the stock market theater could have real consequences.

I figure the chances are small and probably very hard to predict, but it would be interesting to know what's considered acceptable risk for a catastrophic market convulsion: 1 in 10? 1 in 100?

Agreed, that's why I'm just optimistic that we'll work these things out, not "everything's already fine, nothing to see here" :) Anything that forces you to trade without your explicit consent, be it algo or margin call, is a potential danger.

I think a "flash crash" pretty much already is an algo panic. Exchanges have various rules for curbing volatility to minimize such messes, so I'm pretty sure that anything that's caused just by algo behavior should correct itself in short order (as it always has so far). The fundamentals don't change, after all. Algo panic triggering an actual human panic is only marginally different I think... certainly would be bad PR to get blamed for something like that, but at the end of the day if you were investing based on fundamentals, you were either right in which case the panic is a huge opportunity for you, or you were wrong, in which case you're out of luck regardless of the cause. I seriously question the wisdom of doing any retail "investing" based on anything other than fundamentals when you're trading against robots. Or against human professionals, for that matter. I'm not sure how much can be done to protect people that do this from themselves.

Sure I am probably misinformed about how HFT works, but it's obvious that flash crashes are malfunctions and not calculated risks. The last flash crash didn't require taxpayer intervention, but it's not unreasonable to imagine one that would affect innocent people.
Why not damp things a bit with a transaction tax, then lower it gradually once things look more stable?

I was about to make a response with a similar argument and it occurred to me a loss of faith from retail investors, such that total investment dollars are reduced, is not necessarily good or bad as the goals of equity investing in the modern market are largely decoupled from the original goal of providing funding to companies.

Someone less risk averse can step in and seize a previously unavailable profitable opportunity if demand for X is lower or the spread is wider.

Also, the flash crashes were intraday events that were wildly profitable for traders paying attention and wildly unprofitable for the people whose algorithms made mistakes. There was not much diffusion of costs among "innocent" parties. If that discourages mom and pop from day trading, that is arguably a social good as they would be competing with the deck stacked substantially against them and would probably derive more utility from using that money in other ways.

That response was to Apoorv and I took way too long to write it :-)

i just want to point out that massive investment was made in HFT in 2008 when nobody had good ideas where to invest money. A lot of money poured into an area where there are finite profits to be made. Probably net negative for investors. Little will be invested going forward.

What it does is change the stock exchange from a market where all buyers and seller participate, to a market where a few operate. The real markets then become the brokers who buy and sell to all their clients, using the HFT as an interface to the stock exchange.

The diminishing returns of the HFT trading on the exchange will mean that over time the brokers become exchanges in themselves, or rather half a dozen exchanges, each competing to attract clients who would trade among themselves. Over time that number will diminish. Over time the big name exchanges will face competition from these newly formed exchanges who are the ones setting prices because that is where the trading action is.

I suspect we will see a small IPO happen entirely within the confines of one of these firms. And we will see a divergence within these firms of the stock prices from the official exchanges.

my understanding is that the HFTs are frequently there to scalp orders - so instead of allowing two market orders to clear with each other, the HFT algo narrows the spread slightly to capture both orders. Each of the two traders would have gotten better execution had the trade been allowed to transact between them directly.

As to the profiting from HFT algo mistakes, they don't let that happen either as the trades keep getting canceled. Say I have a limit order in to buy AAPL at $500 - I might get executed during a flash crash only to have the trade canceled because it was "clearly erroneous." But if the stock slides and stays down I won't have it canceled. So obviously there is no incentive to put in limit orders below market - and therefore the market loses an important liquidity backstop from long-term, patient value investors.

Your understanding is wrong. By narrowing bid/ask spreads HFTs actually allow "regular" traders to get better prices not worse prices. Let's say that there is a stock with a very narrow bid/ask spread of a penny due to HFT. I can buy at 10.00 and sell at 9.99. In the absence of HFT that spread maybe goes up to a dime. All of a sudden if I want to buy I have to pay 10.05 and if I want to sell I can only get 9.95.

HFT makes it so I can buy for less and sell for more.

If the core purpose of the market is to allow companies to raise capital to invest in new products and services, and in turn reward investors for wisely choosing companies with which to endow their hard-earned money, high frequency trading is so far removed from those things that any talk of a social benefit is meaningless.

Gambling has just as much, and perhaps more, social merit.

You would hopefully acknowledge that Warren Buffet finding a stock priced at $50 that should be valued at $60 has "merit" to it. I.e. even though it's on the secondary market Warren Buffet increases the efficient allocation of capital. He does so because companies looking to raise capital either by IPO or secondary offerings will look at the valuation of similar firms on the market. So if a firm is mis-priced by $10 and Buffet pushes it back in line, that has "social value" because potential investors will look at those valuations. You certainly agree with this, correct?

Now what if Buffer changes his model. Instead of looking for stocks mis-priced by $10, he's going to find stocks mis-priced by $5 and double the number of them he invests in. There's no reason that should be any less "socially valueable" after all he's correcting the same total cumulative mis-pricing. Now take it a step further he'll find stocks mis-priced by $1 on average and invest in 10 times the number of companies. If the tenth dollar of mis-pricing correction is valuable than the first dollar must certainly be just as much, if not more so due to decreasing marginal returns.

Finally Buffet changes his model to finding firms mis-priced by $0.01, and takes 10,000 times as many positions. Buffet now looks like a high-frequency trader. He corrects the same cumulative mis-pricing as before, yet somehow you would claim he adds no social value. If bringing a stock from $60 price to its fair price of $50, then bringing it from $60 to $59.99 must be at least 1/10000 as valuable. Such that if you do that type of trade 10,000 times you're producing the same social value as the $10 mis-pricing corrector who trades once.

That $10 is just made up of a whole bunch of pennies. And if the individual pennies don't add value then by simple induction neither does the $10. So either you must concede that no one who trades on the secondary market adds value, in which case you should be attacking the much larger investment industry, or that HFT adds value.

By your logic, the shaving of milliseconds becomes the shaving of microseconds becomes the shaving of nanoseconds, all of which have social value despite ever-diminishing returns and ignoring any and all costs involved - both real costs and opportunity costs as raised elsewhere in this thread.

the hidden costs of HFT are totally unrelated to the pure economics and arbitrage scenarios you are discussing here, Tyler.

there are two very high hidden costs to this that I think really need to be taken more seriously. First, is the issue of market access. HFT allows heavily invested insiders with proprietary infrastructure to dominate and chase out of the market competing traders who are less invested in this infrastructure. They might rent out their infrastructure, but then they have become rent-collecting middle-men which is a clear inefficiency in the market.

The second hidden cost is the result of an arbitrage strategy. They might simply use their infrastructure dominance to effectively manipulate prices by a series of short term market-cornering trades. This is in fact the usual arbitrage strategy of HFT. The HFT trader collects fractions of the bid-ask gap that would normally go to another buyer as profits. In essence this is disrupting the balance of profits in favor of the market-maker strategy and away from the buy/hold strategy. The end result of this is an extreme increase in market volatility, including the possiblity of events like "flash crashes" and other disruptions of markets. Also a very high cost.

In the old world you had a cartel with high spreads for market-makers, they got preferential access to the trading book, and in return they had an obligation to maintain an orderly market, and commit capital to do so.

Technology and the SEC broke up the cartel. Now you have low spreads, HFT systems get preferential access through low latency. but they have no obligation to maintain an orderly market.

When HFT systems sense that normal relationships are breaking down, they step away and you get a very large discontinuous increase in spreads and drop in liquidity.

HFTs play a lot of bizarre games to get an edge against other HFT systems, constantly changing quotes to obscure intentions and the true depth of the market (quote stuffing).

Trading is probably more efficient, lower spreads, portfolio managers can just use a VWAP or other algorithmic strategy to participate in flow and not need as many traders, or any traders. But HFT profits are quite large, it would be interesting to estimate them compare them to the old cartel. To some extent you just exchanged one set of chiselers for another.

Having a class of participants with preferential access constantly scalping long-term investors doesn't enhance faith in markets (although going back to a cartel is not particularly commendable as a solution).

HFT shenanigans increase the attractiveness of 'dark pools' like Liquidnet, where large portfolio managers do price discovery off-exchange directly with each other for large blocks. That trading is not accessible to HFTs, nor to smaller and retail investors, which limits the public good of the national exchange and public price discovery.

My sense is it would be advisable to incorporate rules that would limit the advantage of HFTs over other participants eg minimum quote duration and size.

Let's apply the kinds of tactics used by HFT to profit in distributed markets to wreckers and ambulance companies and auto body shops broadcasting false vehicle position information into the distributed traffic signalling systems, that Tyler and Alex are so excited about, for profit - by exploiting Byzantine failure modes in the coordination algorithms, crashes occur leading to profitable wrecker, EMT, and body repair business.

That would be considered a crime, but proving it took place would be difficult and require a central authority recording data for later analysis to prove the false information existed and then to prove a pattern of false information.

A second kind of false information would be that broadcast by a car that is trying to move faster than the road and traffic allow if everyone follows the rules. By broadcasting false information, cars in the adjacent lane can be tricked to slow down for a non-existent obstacle, so you can switch lanes and race ahead, this is then repeated for your original lane so you can race ahead in that lane. All the traffic is slowed, possibly to a stall behind you, to allow you to make faster progress that otherwise possible.

Again, this could be prosecuted if the evidence could be found to prove violating the rules of the road. But merely arriving ahead of the rest of the traffic is not a crime.

Guy, I hate to rain on your parade, but the kind of "tactics" you're describing here generally fall under the category of this thing called "market manipulation". That's, um, generally frowned upon. Are you seriously suggesting that the HFT industry is able to make money because it's able to cleverly manipulate markets this way while staying a step ahead of the SEC for the past several years? I assume you must have some evidence to back up such a claim.. maybe you're a whistleblower who works or has worked for an HFT firm and is about to blow this whole thing wide open.

One alternative hypothesis also comes to mind, but I'll be polite and let you fill in the blank on your own

"HFT shenanigans increase the attractiveness of ‘dark pools’ like Liquidnet, where large portfolio managers do price discovery off-exchange directly with each other for large blocks. That trading is not accessible to HFTs, nor to smaller and retail investors, which limits the public good of the national exchange and public price discovery."

The primary reason that dark pools are popular is because they're exempt from the penny increment system. I.e. you can trade at sub-penny prices there, whereas at exchanges you cannot. So for stocks that have thick books, i.e. constantly have 1 tick spreads, you can get a better price in the dark pool. This is just completely arbitrary regulation by the SEC. If lit exchanges were allowed to offer sub penny prices dark pool volume would collapse.

"HFTs play a lot of bizarre games to get an edge against other HFT systems, constantly changing quotes to obscure intentions and the true depth of the market (quote stuffing)."

You're mis-using "quote stuffing," which implies sending a lot of orders to a matching engine to slow down its processing speed. This practice is highly monitored and discouraged by the exchanges. No serious player does it because the exchanges would ban your firm for life. Not only that but the exchanges have software that will simply drop your connection if you try to do this. In fact there's no documented case of anyone actually trying this.

I don't know what you mean by "obscuring the depth" of the market but I guess you mean posting large bids or asks to move the market. Like I mentioned above HFT makes this harder not easier to do. Computers can analyze huge datasets to figure out the characteristics of such gamed quotes. When strategies detect them they'll be more than happy to cross the large posted liquidity, thus neutralizing the order and forcing the gamer to suffer a loss.

As for quote cancellation, this is highly mis-understood. The reason that quote cancellation rates have risen is because market makers use computers to real-time adjust their quotes against the referential price movements of more liquid security. So if I'm posting liquidity in a mid-cap stock, SPY might tick 20 or 30 times before any orders come in. I'm going to keep modifying my quote price based on those ticks. It will look like a bunch of orders keep coming in, then shortly after get cancelled. In reality this represents a single "meta-order."

"But HFT profits are quite large, it would be interesting to estimate them compare them to the old cartel."

HFT profits are larger because volume is orders of magnitude higher than in pre-electronic days. By lowering trading costs HFT made it possible for massive increases in transactions. So there's far more consumer surplus being generated.

Dark pools are popular because large blocks can be negotiated without public price discovery. If a large seller tried to advertise 1m shares in the public order book, it would impact the price. The dark pool lets someone say I have 1m shares I'm willing to part with at X price, and the order book is secret, but if someone comes along willing to buy at that price, they get matched up and the trade prints (and in some systems they can negotiate anonymously by instant message). It's ignorant or foolish, to think that if the public order book went to sub-penny increments, the dark pools would go away.

Also anyone who's looked at a quote stream can see there are funny patterns that seem far too regular to be accounted for by coincidental moves in related markets, and anyone who's played poker knows that you manipulate your bets to hide information.

HFT is not just about trading faster to make a more efficient arbitrage, which as Tyler's points out, is fine theoretically. The rent-seeking element of HFT is that it's basically front-running. While it's curtailed somewhat recently because of scrutiny, some dealers/market makers give traders deeper access to order books, or soften their Chinese Walls for their own traders (e.g., Knight executes orders for clients, but the same group also trades for its own book). An additional HFT strategy is to put out an excessive amount of fake orders, enticing other traders to bid on them, but canceling them before execution is completed (this is where the faster part comes in) in order tease out genuine order intentions or to distort short-term prices in a certain direction.

HFT has become a catchall term for various strategies, some of which are more socially damaging than others. Things like flash crashes gain more attention, but are less harmful in the long run.

"Knight executes orders for clients, but the same group also trades for its own book"

Then don't give your orders to Knight, why is that so hard? You can route your orders wherever you want them to go and you can hide your orders if you don't want them on the book.

Some large investors do route their trades outside of the exchanges using dark pools, which avoids HFTs, but this does have other inefficiencies (market is much thinner). Smaller investors don't have access to dark pools and generally have to route their trades through an exchange market maker (who execute on behalf of TD Ameritrade, Schwab, E*TRADE, et al.). Avoiding the large market makers is easier said than done, and my guess is that individual investors as a whole aren't aware they're paying this toll and thus aren't clamoring for better execution from their brokers. As for "hiding" your orders, investors have to route trades through a broker somehow; the issue is how protected the trade information is from other parties.

Best case scenario HFT provides the same benefits to society as a professional poker player.

Worst case scenario HFT erodes all confidence in public financial markets.

I believe some skepticism is warranted. At some point stocks simply become part of a meta-game of algo traders with no relation to any fundamental value.

Let me talk about HFT from the standpoint of distributed computing system design.

The problem is the same as Byzantine Generals' Problem. From wikipedia:
"Byzantine refers to the Byzantine Generals' Problem, an agreement problem (first proposed by Marshall Pease, Robert Shostak, and Leslie Lamport in 1980)[3] in which generals of the Byzantine Empire's army must decide unanimously whether to attack some enemy army. The problem is complicated by the geographic separation of the generals, who must communicate by sending messengers to each other, and by the presence of traitors amongst the generals. These traitors can act arbitrarily in order to achieve the following aims: trick some generals into attacking; force a decision that is not consistent with the generals' desires, e.g. forcing an attack when no general wished to attack; or confusing some generals to the point that they are unable to make up their minds. If the traitors succeed in any of these goals, any resulting attack is doomed, as only a concerted effort can result in victory."

The HFT are acting as "traitors" in the hopes of making a trade, "forcing an attack", based on a "general" deciding incorrectly from the market agreement of all the "generals".

The HFT consider the coordinated action of all the real market participants to be contrary to their profit, so the traders see any way possible to exploit the distributed nature of the modern securities market.

In the original market place, the market maker provided all the market coordination, evolving over time into a speculator with slightly superior knowledge to profit, but every market participant saw every bid offered and completed and could with effort track the market maker position. The software and hardware system developers have tried to replicate the trading floor, but without a single "floor" through which every trade must flow. This is an extremely complicated problem that must trade perfection with throughput without livelock or deadlock.

The HFT have in the past decade been targeting the inherent faults of the pragmatic distributed computing solution which has inherent Byzantine failure modes. They are sending in traitors to trick generals into errors of coordination - getting a general to sell at $500 when the market price is $505 based on a $505 bid placed, but not yet received, by the selling general.

Taking an analogy from complexity theory, the discrepancy might not really be limited to p x q as you slice time into smaller and smaller increments. Similar to the question of "how long is the coastline," the answer depends on the scale at which you measure. (The coastline is essentially infinite if you measure at nanoscale.) Hence, slicing the market into smaller and smaller increments may lead to larger and larger total discrepancies.

Good traders buy low and sell high, lowering overall volatility.
Bad traders on the other hand, increase volatility, selling low and buying high... but also LOSE money in doing so... so whats the problem?

This is an unavoidable fact regardless of the situation.
For every trader exacerbating a big move, there is another (profitable one) reducing the move.

Don't forget the costs of regulating, auditing, etc. Why should all of us subsidize a niche business ? And it distracts regulators from other, bigger problems, which has its own cost.

The social BENEFITs are high. The cost of trading for mom and pop is far lower, far less money is taken out by HFTs than by traditional old-school market-makers (everyone forgets that SOMEONE makes money on fascilitating trading, they just yell and scream at whoever is making it today). But, luddites and semi-socialists and Congressmen (and the considerable overlap therein) know this is hard to explain, or don't understand it themselves, so they have someone knew to yell at. But, hey, progress always has its enemies.

Oh, the other people yelling about this are old-school firms who wanted to fleece people for more themselves. They are probably the worst of the lot. Not stupid, just thieves.

Comments for this post are closed