Who sets securities1 A security is a financial instrument that represents ownership, a creditor relationship, or rights to ownership in an entity. It can take various forms, including stocks, bonds, or options, and is typically traded on financial markets. Stocks (equity securities) signify ownership in a company and a claim on a portion of its assets and earnings. Bonds (debt securities) represent a loan made by an investor to a borrower, usually corporate or governmental, with the promise of repayment with interest. prices, and how? These simple questions—and their answers, which are more complex but in some ways just as straightforward—often get lost in the reporting on financial markets. Whether the “line goes up” or “line goes down” on a stock chart, we rarely stop to ask, “Wait, what even is this ‘line’, at a fundamental level?”
When you Google the current market price of Apple or Nvidia, and it shows you an amount of currency per share of stock, what does that price mean in a literal sense? Who sets it? What causes it to change? Within the scope of financial markets, these quantities have specific meanings. Yet when they’re discussed among the broader public or in the business press, the experts’ explanations often feel like a physics professor describing particle interactions by saying an electron “wants” to be repelled when it encounters another electron – that is, in a way intended to be pedagogically useful, though clearly not the full story. So you’ll often hear arcane phrases about how “the market price of a stock is determined by the market forces of supply and demand,” about how its price was “pushed up” by there being “too much supply relative to demand,” about how it’s “overvalued” and “due for a correction,” and so on.
If you’ve ever read or heard statements like these and thought, “Okay, sure…but what exactly do you mean by ‘pushed’? What is actually happening when this occurs?” – then this essay is for you.
Frederic S. Lee, a heterodox economist known for his significant contributions to price theory for industrial firms, emphasized that, fundamentally, prices are directly administered by firms according to a cost-plus markup equation; his research drew upon many foundational surveys of business owners who were asked how they set their prices.2 See John Michael Colón, “Wobbly Economics – PaaI” in Strange Matters Issue Two (Spring 2023), as well as my essay “Notes Towards a Theory of Inflation” in Strange Matters Issue One (Summer 2022) He noted that these firms are generally hesitant to raise prices for fear of losing customer goodwill, and as a result, markets for their goods exhibit a surprising degree of pricing discipline (or stability) over time – in contradiction to standard neoclassical economic theory, which posits that a market which is operating efficiently will adjust prices significantly more readily to changes in supply and demand conditions than observed reality would suggest. Ultimately, this research led him to totally reject orthodox microeconomic theory and its concepts of equilibrium prices, marginalism, price discovery, and so on; instead, he replaced these with his own model based on the theory of administered prices.3 See Frederic S. Lee, Post-Keynesian Price Theory (1997) and Microeconomic Theory: A Heterodox Approach (2018).
During a lecture on the topic of price theory,4 See (in John Michael Colón’s nomenclature) Lecture 6 starting at 08:00 and Lecture 9 starting at 42:00. These are unfortunately a privately recorded session of Lee’s graduate economics course at University of Missouri Kansas-City. Audio available upon request. a student asked Lee if his theories on price administration extended to asset markets, such as the New York Stock Exchange (NYSE). Lee acknowledged that, in his estimation, these prices were a different beast: while their fluctuations might sometimes resemble the predictions of neoclassical economists, they were actually governed by what he called “administered rules.” What he meant by this is a bit vague, but he seemed to mean that the operational rules of securities exchanges, as institutions, acted similarly to an auctioneer setting prices. Rules of this sort, he seemed to think, would dominate or at least significantly guide the pricing procedure of the relevant markets, and perhaps account for their unusual behavior compared to the rest of the economy.
The exact nature of Lee’s auctioneer (as well as the dynamics that might result from such a construct) was left unspecified. When we examine the price dynamics of securities on exchanges like the NYSE, we encounter a far more complex and intriguing reality – one whose fluctuations demand to be understood on their own terms. Although the exchanges themselves and their operational needs as businesses play a role in some price-setting processes, for the most part they actually serve a supporting function to the market participants who use them as trading venues – individual traders, institutional traders, market makers – and it is these actors whose trades directly determine price formation.
Church and Wall Street, New York City.” Published 1909 by The American
Art Publishing Co., New York City. #R-43919. The back reads,
“The New York Stock Exchange, located on Broad Street,
with an entrance on Wall Street, is built entirely of carved white
marble. It was founded on May 17, 1792, the present building
was finished in 1903. The board room is 112 by 138 feet and
80 feet high with the ceiling in gold relief. There are 1100
members trading daily from 10 A. M. until 3 P. M.” Wikimedia
Commons, Public Domain.
However, as we will see, while traders ultimately administer prices by submitting their orders to trading venues, they do not do so entirely at their discretion. Contrary to the views of neoclassicals and, to some extent, Lee, the reality of price formation in financial markets is far stranger and more complex than most theorists have envisioned. Yet, out of this complexity, financial markets exhibit a surprising resilience and a fine structure, reminiscent of the price-administrative operations in industrial firms. All this is well worth exploring. It turns out that the relatively high degree of volatility and the clustered, often unpredictable trading activity characteristic of these financial markets give rise to important statistical regularities that offer clues about the pricing procedures underlying them at the micro level.
Starting in the late 1970s and continuing to the present day, a little-known subfield of economics known as market microstructure has focused on the microfoundations of financial markets, moving away from many mainstream financial economic models like the efficient-market hypothesis5We shall return to this concept later on. in favor of models better-grounded in observed phenomena. This empirical approach led market microstructure researchers into uncharted territory, where access to significantly better data at the turn of the twenty-first century led them to a structural break from most of the rest of mainstream economics. These dissident neoclassical economists (a phrase I never thought I’d write, by the way) essentially forced a Copernican Revolution for the field. This choice made by the sub-discipline starkly contrasts with the broader discipline of economics, which often ignores data that contradicts its theories.
My goal in what follows is to cut through the jargon and focus on the concrete, measurable realities of how prices are set and evolve in financial markets, to reformulate the vague and often misleading notion of equilibrating supply and demand curves yielding some market price into a more accurate and precisely defined price-administration process driven by trading activity itself. By the end of this exploration, I aim to develop a theory of price formation that not only respects but is firmly grounded in the well established empirical facts of financial markets – in keeping with the laudable empirical turn of the market microstructure economists.
So, grab a cup of coffee, settle in, and grab a microscope, because we’re about to take a close look at how the formation and evolution of security prices depend primarily on the price-setting actions of individual market participants – and how out of this complex process, securities prices are administered.
Part I:
Limit Order Books:
Their Mechanics and Basic Dynamics
In order to prepare you conceptually for the theoretical argument that follows, we need to take a trip through the basic elements of financial markets.6 In the section that follows, my description of financial market operations is a synthesis for laymen of various technical sources. The most important of these, and the best introduction for anyone who wants to dig deeper, is Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, & Martin Gould, Trades, Quotes and Prices: Financial Markets Under the Microscope (2018). Another good source is Daniel Spulber, Market Microstructure: Intermediaries and the Theory of the Firm (1999). Where I felt it would help readers to have sources for the definition of specific terms, I’ve added additional footnotes with page reference to Bouchaud et al and Spulber, or with references to articles from common authorities like Investopedia. Although you’ve likely heard of each of these terms at least in passing when a bit of financial news flashes across the screen or appears in your social media feed, it’s important to understand each of the terms and how they fit together in the real operations of financial markets. Please note that while I’m going to be using stock markets for our examples, these concepts apply broadly across markets for all types of securities (including commodities, cryptocurrencies, bonds, forex, derivatives, and much more).
Basics of the Book Itself
Orders, limit orders, and market orders
Let’s start with the most elemental building block of any financial market: the order. What is an order? It is a commitment to buy or sell a given quantity of an asset at some given price. There are four required attributes for an order: its sign (is it a buy or a sell – often denoted with a + or -, respectively), its price (quantity of a currency to be paid), its volume (a quantity of the chosen security), and its submission time (when the order was successfully submitted into a queue at a trading venue).7 This could be an exchange like the New Stock Stock Exchange (the type of trading venue we’ll be discussing most in this essay). But there are other types of venues as well, with one major type being alternative trading systems (ATSes). To help you remember the attributes, please refer to this succinct notation:
x := (s, p, v, t)8 Bouchaud et al, p. 45.
Where the order x has its sign ‘s’ (+ or -), a price ‘p’, a volume ‘v’, and a submission time ‘t’. There are many different types of orders one could make, but all of them need at least these four attributes in order to be accepted into an order queuing system. The two most common types of orders (and the ones we will be referring to most frequently in this piece) are limit orders and market orders.
Limit orders at their most basic level are orders submitted with a specified price that is the worst price (the maximum for buyers, the minimum for sellers) the trader who set it is willing to accept. These orders signal a specific price intention, and the traders who submit these orders are essentially saying they’re willing to wait if they have to in order to get their preferred price (or better). Market orders, on the other hand, are orders that are submitted with no specified price – traders who submit these orders are signaling that they favor speed of execution over price. For all practical purposes, you can think of it this way: limit orders have prices set by market participants, whereas market orders do not initially have prices because participants who place them care more about speed.
You might be asking yourself: if market orders don’t have prices, isn’t that a violation of the basic conceptual order attributes that were just mentioned? The answer is no, as they are still technically submitted with a price – it’s just set to null. That said, a market order does eventually acquire an actual price – when it is matched by the financial market’s algorithm to a limit order of the opposite sign. This is called order execution, or filling an order.9 Investopedia. “Execution.” Investopedia. Last modified August 17, 2023. https://www.investopedia.com/terms/e/execution.asp. The price that any market order ultimately gets executed at is always a price that had been set not by “the market” but by some other market participant, a counterparty whose limit order has been matched to this market order – and so the market order (which had a price of null) is ultimately executed at the price of the limit order to which it was matched. Hence, the answer to the question, “who sets prices on financial markets” is always, eventually, “market participants who set limit orders to buy or sell at a particular price” – though it takes matching with a market order for that command to be carried out.
When an order executes on the market, what it means is that a market order (say, to buy) at the null price has been matched with a limit order (say, to sell) at some particular price, so that the seller gets the buyer’s cash and the buyer gets ownership of the security (say, a stock). The relationship between this operation and the price of the security is relatively straightforward. When you see a price for a particular stock, option, currency, etc. quoted (e.g., “TESLA is at $250 per share”), or a graph of those quotations over time, this is usually what’s called the market price. The market price is quite simply the price of the last executed order – the price of the last limit order somebody placed that matched with a market order and was completed.10 Investopedia. “Market Price.” Investopedia. Last modified May 24, 2023. https://www.investopedia.com/terms/m/market-price.asp. Simple as it is, this principle will have some interesting implications for theory – especially any theory that wants to believe the market price is some sort of optimal supply-demand equilibrium. In fact, all you’re seeing in any given market price is nothing more than the history of its previously executed orders. Any theory of financial market prices must be a theory of the flow of orders that runs through them – how and why they’re placed, and the ways each order potentially affects subsequent ones.
Limit Order Books and their Mechanics
Modern financial markets rely upon limit order books (hereafter referred to as the “LOB”) to keep track of all the limit orders – the orders with prices set by market participants – that have so far been submitted. Each security has its own LOB – this ensures orders for one security are not mixed in with orders for a different one. You can think of it as a list of all of the limit buy orders (called Bids) and limit sell orders (called Asks) collected together and sorted in descending order from best to worst.11 The Bid price and the Ask price at the top of the LOB at any given time – the first row in any of the tables in my examples below – are often displayed together on trading apps as the asset’s current best Bid and Ask price. The Bid is always kept lower than the Ask, for reasons to be explained in the next footnote below. This creates a gap between the top Bid price and that top Ask price, which may be wider or thinner depending on which limit orders have been placed into each column of the book and how market orders have chewed through them at any given moment. The difference between the Bid and Ask price at the top of the LOB is called the bid-ask spread. “Best,” that is, for a prospective counterparty – the LOB prioritizes the limit orders with prices most favorable to the opposing side.12 In other words, to spell it out a bit more: any limit buy order or bid is the highest price that a buyer (who prefers to buy low) is willing to pay; but the limit order book will prioritize the one that’s highest and execute that first, which is in the interest of the seller who puts in a market order. Similarly, any limit sell order or ask is the lowest price a seller (who wants to sell high) is willing to accept, but the limit order book will prioritize the one that’s lowest and execute that first, which is in the interest of the buyer. In other words, LOBs will keep participants who place limit orders waiting in order to provide participants who place market orders the best deal; they will only execute limit orders in the sequence that’s most generous from the perspective of the opposite party. Why do this? Arguably, from the point of view of the operators of financial market infrastructure, it’s in their own self-interest to do so. A market that gives counterparties the best possible deal given a certain LOB state is one that will result in more matches and executed orders; more executed orders means not only more commission fees for the exchange, but also more happy customers, and hence quite likely higher trading volume and hence more commission fees. The main business model of an exchange is to get as many people trading on it as possible. So the best Bid (the limit buy order that has the highest price) is at the top of the Bid side of the LOB, with successively worse Bids listed underneath it; and the best Ask (the limit sell order that has the lowest price) is at the top of the Ask side, with successively worse Asks listed underneath it.
Modern securities exchanges (like the New York Stock Exchange, the NASDAQ, The London Stock Exchange, and others) deploy matching engines – algorithms that execute trades in LOBs according to a fixed set of rules. Each LOB has its own instantiation of a matching engine. The most common type of matching engine ruleset is price-time priority,13Most exchanges use a vanilla price-time priority matching engine, which is a linear optimization favoring price over time. However, some exchanges, like the NYSE, use a slightly different approach called price parity allocation. This algorithm is similar but adds a term for “par pricing,” where orders at the same price level are aggregated, and the size of each order is adjusted proportionally to ensure fairness among participants. This means that if multiple orders exist at the same price, each may be partially filled based on the available volume, with the remainder left on the book. in which orders are prioritized first by price and only then by the time they are received. This means that the highest buy orders and the lowest sell orders are matched first, and among orders at the same price, those that arrived earlier are executed before later ones.14 In the examples that follow, whose purpose is to introduce readers to basic limit order book operations, we’re treating the algorithm as if only price mattered to it. In real life, however, matching algorithms of the common price-time priority type consider price first, but time second. This means they’ll try not to keep market participants waiting if they can help it, but prioritize getting the best price first. That said, the general principles are the same, since we’re going to be considering the most important component, price.
Now, imagine your favorite (or least favorite) company’s stock being traded on a financial market for $x/share. Let’s walk through a few examples where orders of each sign are executed, resulting in fluctuating market prices for the stock and an evolving limit order book.
We begin with a simple example of a LOB where a market order interacts with limit orders. This example will illustrate how a market order affects the order book and influences price dynamics.
The market price (the last-executed price) starts at $50.15 From here on out, I’m just going to list the market price without clarifying it’s per share, for purposes of simplicity. So “$50” instead of “$50/share,” etc. Whenever you see a stock price, you should add the “per share” back in your head. In any given transaction, the actual amount of cash being paid is the share price x the number of shares. (e.g., “sell 300 shares at $100” if executed nets the seller 300 x 100 = $3000!) So just for fun, keep an eye on our first market order. How much money does the seller make once it’s executed? How much does the buyer spend? (Hint: are these the same? Think carefully!) I’ll have the answer for you in another footnote once the transaction is complete. It won’t be long until a market buy order comes in. But before the market buy order, the limit order book looks like this:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 100 shares, $51.00 | 50 shares, $51.50 |
| 50 shares, $50.50 | 200 shares, $52.00 |
| 200 shares, $50.25 | 100 shares, $52.10 |
Remember what you’re looking at in this table: the Bid side is all the limit buy orders, the Ask side all the limit sell orders placed by market participants, at the price they desire. None have been executed yet. When they are executed, by matching with a market order, they are removed. But these limit orders never interact with one another. Rather, for a limit order to be executed, a market order of the opposite sign must come in and be matched to it by the matching algorithm. (So for example, a limit Bid order to buy 100 shares at $51 will never be executed unless a market order to sell comes in. Similarly, a limit Ask order to sell 50 shares at $51.50 will never be executed unless a market order to buy comes in, etc.) Let’s run through an example to see what happens when that order does arrive.
A market buy order for 200 shares hits the LOB. Remember, market orders always match to limit orders of the opposite sign – in this case, to sell orders on the Ask side. Let’s see what happens to the order book after the market buy order:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 100 shares, $51.00 | *Filled: 50 shares, $51.50* |
| 50 shares, $50.50 | *Partially Filled: 150 shares, $52* 50 shares, $52 |
| 200 shares, $50.25 | 100 shares, $52.10 |
The buy order for 200 shares moves down the Ask column from top to bottom. It first fills the 50 shares at $51.50, chewing through the whole first box; it then fills the next 150 shares at $52. There were originally 200 shares in the $52 limit order (the second box down the column), so 50 are left over after the market order eats through the rest (200 – 150 = 50); these remaining 50 shares are left at the top of the limit order book, with the same $52 price. Meanwhile, the stock’s new market price is $52 because that’s the price of the most recently executed trade – that is, $52 is the limit price of the last filled component of the trade we just executed.16 And now comes the answer to my quiz question in the previous footnote. Did you figure it out? Part of the trick is that each limit order is (probably) placed by a different seller. So the first seller, whose 50 shares were all consumed by the order at $52/share, made 50 x 51.50 = $2,575 from the transaction. The second seller, who sold only 150 of the 200 shares they wished to sell for $52/share, made 150 x 52 = $7,800 and still has 50 shares to sell. This means the buyer, to acquire their 200 desired shares, spent a total of 2,575 + 7,800 = $10,375. This is a good exercise to repeat for each of the transactions below! Also note that the very same single large order can experience execution at very different prices as it chews through the order book. When a single order is executed at multiple price levels, market practitioners often take an average of these different levels, weighted by the number of shares at each price level, to gauge their performance. Basically, you take the money total and divide it by the number of shares ordered. This is called the average execution price. In this case, the average execution price was [ (51.5 x 50) + (52 x 150) ] / 200 = $51.88. The average execution price for the same security can vary significantly depending on the composition of the limit order book.
Here’s how the new LOB will end up looking:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 100 shares, $51.00 | 50 shares, $52 |
| 50 shares, $50.50 | 100 shares, $52.10 |
| 200 shares, $50.25 |
So much for a buy order. Now let’s look at an example where a sell order comes in. Remember, it will interact with the limit orders of the opposite sign – in this case, with those on the buy side.
A market sell order for 200 shares arrives, filling the first two levels of the Bid side of the LOB, and partially filling the third box as well:
| Bid (Buy) | Ask (Sell) |
|---|---|
| *Filled: 100 shares, $51.00* | 50 shares, $52 |
| *Filled: 50 shares, $50.50* | 100 shares, $52.10 |
| *Partially Filled: 50 shares at $50.25* 150 shares, $50.25 |
After the order is fully executed, the stock’s new market price is $50.25 (the price of the last batch used to fill the order – i.e., the third box down the left column of the LOB). The new LOB looks like this:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 150 shares, $50.25 | 50 shares, $52 |
| 100 shares, $52.10 |
There are a few interesting differences worth pointing out between this and the last order we examined. The most obvious is that the price went down instead of up – from $52 to $50.25 – because market sell orders interact with limit buy orders, which are arranged from highest to lowest to benefit the seller. Hence, as a market sell order eats through the available limit orders in the book, it arrives at prices that are lower the deeper it gets – the opposite of a market buy order, which encounters higher limit prices the deeper it gets.
Additionally, and more subtly, the proportions don’t exactly line up how you’d expect. Both the earlier market buy order and this market sell order were for 200 shares each. Yet the former chewed through 1¾ boxes in the book, while the latter chewed through 2½ boxes – because the orders at the top of the LOB’s Ask column were larger, i.e. had more shares in each box, making the same-sized 200-share market order get through fewer boxes. What’s more, while both moved the stock’s market price, it was not the one that chewed through more boxes and passed through more prices that affected the price more. The market buy order (which ate up only 1¾ boxes) increased the stock’s market price by $2, while the market sell order (which ate up 2½ boxes) only decreased the stock’s market price by $1.75. Thus, there is no strict functional relationship between the size of a market buy or sell order and its effect on a market price, or vice versa. Rather, the characteristics of the limit orders placed into the limit order book determine how any given market order will, or won’t, affect the price. The number and size (i.e. number of shares) of the limit orders in the LOB determines how big a market order must be to eat through more than one box and so arrive at a new market price; this is called the depth of the order book. All other things being equal (i.e., restricting ourselves to a purely static case), a shallow order book will see more price changes, a deep one fewer, with the same market orders. Finally, while the prices set by market participants who place the limit orders in the book are always ordered best to worst, these prices may be closer or farther from each other depending on how they happen to have been set. If the prices in the limit order book have larger gaps between them, there will be bigger price changes as a market order chews its way from one box to the next within the book.
Note the relationship between the market order, the LOB, and the stock’s market price. The market buy order comes in; it starts with the lowest price on the Ask column, this being advantageous to the buyer; having exhausted all the shares at that price, it goes for shares at the next-best price, some amount higher, working its way down the limit order book in this manner; and when the buy order is completely filled, the last price it was filled at becomes the stock’s new market price. In this case, a single transaction bumped the stock price up from $50 to $52. The opposite was true for the market sell order, resulting in a stock price bumped down from $52 to $50.25. In general, it’s the fact that (limit) buy and sell orders in the LOB are executed in an order most advantageous to their counterparties which explains why (market) buy orders tend to increase and (market) sell orders to decrease the market price of securities. (Tend to – but not always, as we’ll see in a coming example.) Thus, the structure of the LOB and its interaction with market orders are what causes fluctuations in the prices of securities in financial markets.17 This may seem like an obvious enough point in the context of Mann’s practical, concrete explanations in this section. But the empirical fact that prices on financial markets fluctuate so much with new orders was one of the last phenomena left unexplained by the theory of administered prices, the core component of the alternative microeconomics with which the heterodox economist Frederic S. Lee proposed to replace supply-demand equilibrium models. The administered price theory describes the vast majority of prices in the economy, which are set by producers as a mark-up over cost and remain stable irrespective of demand; but since this left unexplained how financial market prices are administered, and why they’re so volatile, it provided a sort of “God of the gaps” excuse for neoclassicals who want to say that cost-plus prices are an anomaly (despite being the vast majority of prices) and that supply-and-demand equilibrium prices are to be found in ‘pure’ form in these other, more perfect markets. As it turns out, the superficial similarity of reality to neoclassical theory – the fact that securities prices sometimes (not always) move with new orders – belies a deeper disjuncture, since this movement has nothing to do with a process of price discovery where sellers’ decision to sell and buyers’ decision to buy are nothing but a functional reaction to some price they receive from “the market,” and prices therefore produce an equilibrium between supply and demand. Rather, as in the rest of the economy (albeit by a rather different institutionally structured procedure), it’s people who set prices, not the market; they do so according to different strategies, depending on their position and goals; the limit order book matches bids to asks in a manner most advantageous to counterparties according to one or the other algorithm’s priorities; but there is no such thing as an equilibrium price, and prices reflect not some ‘objective’ value they ‘ought’ to be but rather nothing more than the history of previous orders executed by the system; meaning that, as Mann’s essay will go on to explain and as he will explore in subsequent work, financial markets aren’t vast information processors or calculators of real economic value so much as they are casinos driven by bandwagon effects and shared hopes of future gain, whether real or imagined. In short, with Mann’s explanation of the limit order book, the last bastion of neoclassical orthodoxy has been stormed and torched to the ground. For more on this, see Part II of John Michael Colón’s “Wobbly Economics,” forthcoming in the magazine. –Eds.
Order Flow
Order flow in financial markets refers to the continuous introduction of new buy and sell orders that traders submit to a trading venue. It represents the aggregate of all trading activity, showing whether and at what rate traders are willing to buy or sell a particular asset. Order flow includes details such as the size of each order, the type of order (like market orders, limit orders, or stop orders), and the time of execution, which are crucial for understanding market liquidity, depth, and the intention of market participants. Market makers and traders often study order flow to gauge short-term price fluctuations and to make informed trading decisions, as it can reveal some of the underlying strength or weakness in the market of a given security.
Picking up from where we left off in our examples, let’s complicate things further. Suppose that immediately following the above, another large market sell order for 200 shares comes in.
| Bid (Buy) | Ask (Sell) |
|---|---|
| *Filled: 150 shares at $50.25* | 50 shares, $52 |
| [NOTE: 50 shares of the market order remain to be filled!] | 100 shares, $52.10 |
In this rather extreme example, the entire Bid side of the LOB is filled, leaving no liquidity on the Bid side. And yet 50 shares from the market sell order remain unfilled. Hence the rest of the market sell order sits in queue, awaiting new limit buy orders (Bids) to match with.
Let’s say that buyers who had been sitting on the sidelines observing the past 10 minutes of trading take notice and sense an opportunity to buy at a good (read: low) price.18 That is, they believe that despite the fact there’s a selloff currently underway, the security’s market price will soon appreciate, meaning that now is a good time to get a cheap deal since they expect it to increase in price soon enough to yield a nice profit. This practice is called buying the dip. That said, like all trades it carries risk – in this case, that you aren’t actually at or near the bottom of the dip, and that the security’s market price will only continue to decline and not actually increase anytime soon. Presumably the participants who initiated the selloff in the first place are under just such an impression and wanted to cut their losses, getting out of their position while they still could. Who’s right? Well, who cares? (They do, since they’ve put money down on the question, but what about us?) The more important point for our purposes is that, no matter who’s right, the answer has less to do with the inherent value of the security, necessarily, than it has to do with the limit and market orders participants will place in the next day or week – since it is this which generates the interaction between the LOB and market orders that causes prices to move. Those decisions by market participants may reflect the value of the security – or merely their impressions about its values. Or, it may be decided by entirely different factors – perhaps different ones, depending on the participants and their strategies. They could, for example, just as easily be trying to predict each others’ behavior, something I’ll be discussing later in the essay. These buyers replenish liquidity by placing limit buy orders, refilling the Bid side of the limit order book with three new limit buy orders:
| Bid (Buy) | Ask (Sell) |
|---|---|
| *New Order: 50 shares, $49.75* | 50 shares, $52 |
| *New Order: 100 shares, $49.50* | 100 shares, $52.10 |
| *New Order: 100 shares, $49.00* |
This liquidity refill19 Also sometimes referred to as liquidity provision. For an in depth treatment of this phenomenon and some of its theorized motivations among traders, seek out Danny K. Lo, Anthony D. Hall, “Resiliency of the Limit Order Book,” Journal of Economic Dynamics and Control Volume 61 (December 2015), pp. 222-244. enables the remainder of the unfilled market sell order to be filled at the new bid prices. And indeed, some of the new buyers’ orders are executed immediately against the remaining unfilled market sell order:
| Bid (Buy) | Ask (Sell) |
|---|---|
| *Filled* | 50 shares, $52 |
| 100 shares, $49.50 | 100 shares, $52.10 |
| 100 shares, $49.00 |
The new LOB would thus look like this:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 100 shares, $49.50 | 50 shares, $52 |
| 100 shares, $49.00 | 100 shares, $52.10 |
The new market price is now $49.75.
Lastly, let’s consider an example (leaving off from the previous one, as usual) where two things are happening: first, a new limit sell order for 300 shares at $49.75 arrives:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 100 shares, $49.50 | *New Order: 300 shares, $49.75* |
| 100 shares, $49.00 | 50 shares, $52 |
| 100 shares, $52.10 |
Next, a new market buy order for 200 shares comes in. The buy order is matched with the best-available Ask, which happens to be the new limit sell order that arrived only a moment ago. Let’s see how the order book handles this one:
| Bid (Buy) | Ask (Sell) |
|---|---|
| 100 shares, $49.50 | *Partial Fill: 200 shares at $49.75* 100 shares, $49.75 |
| 100 shares, $49.00 | 50 shares, $52 |
| 100 shares, $52.10 |
The result is a commonplace one: despite a rather large market order, the stock’s new market price ends up completely unchanged after the transaction, remaining at $49.75. This is because the limit order at the top of the LOB’s Ask column was large enough to completely absorb it.
Under different conditions, the price effect of the transaction might have been different – a small increase, or even a very large one. After all, in our very first example we saw a market buy order identical to this one that moved the stock price upwards after execution by $2! But here, the same market buy order left the stock price exactly where it was.
Once again we can see that the market price of a security is generated by the interaction of the limit prices set by some participants in the LOB with unpriced market orders set by others outside of it, which are matched by the algorithm according to its (usually price-time) priorities. This system does not particularly lend itself to notions of these prices tending towards any equilibrium whatsoever. However, it does suggest the possibility – indeed, the commonality – of nonlinear and stochastic dynamics in price movements that could suddenly, after a period of stasis, launch them in unexpected new directions.
Liquidity
Having explored the concept of order flow and examined some practical examples of how trades are directed and executed in financial markets, we can now turn our attention to the fundamental notion of liquidity. In order-driven markets, liquidity is intricately tied to the depth of priced orders available at various price levels in the LOB, which (as we’ve seen) directly influences the ease with which trades can be executed without significantly impacting prices.
At first glance, the section header above might feel like an overly basic one. It’s just the amount of money in a system, right? Well, liquidity in financial markets as conceived by market microstructure theorists means something a little different than, say, liquidity in the banking system, or elsewhere.
Remember, financial market operations run on orders, and those need to be supplied by market participants at some price chosen by the participants. Liquidity in market microstructure research refers to the ability of a market to absorb large orders without a significant impact on the price of the asset being traded. It is a crucial concept, as it affects the ease with which participants can execute trades, especially in financial markets where large volumes of assets are exchanged daily. High liquidity implies that there are enough priced orders in the LOB on both sides of the book, allowing transactions to occur quickly and with minimal price fluctuations. Conversely, low liquidity can lead to greater price volatility, as large trades can move prices significantly, making it more difficult for participants to execute their orders at desired prices.
In the context of an LOB, liquidity is often understood as the depth of priced orders – which, you’ll recall, means the number and size of buy and sell orders at various price levels. The depth of the order book indicates how much volume is available at each price point, and thus, how much the price might move in response to a large order. A deep order book, with substantial volume at many price levels, suggests high liquidity because it can accommodate larger trades without causing significant price changes. This depth is critical for traders, particularly those executing large orders, as it reduces the likelihood of slippage – where the final execution price deviates from the intended price due to insufficient liquidity.20 To elaborate: in the context of trading, slippage refers to the difference between the expected price of a trade and the actual price at which the trade is executed. It typically occurs in fast-moving markets or when a large order is executed in a relatively illiquid market, causing the order to be filled at different price levels. Slippage can be positive (better execution price than expected) or negative (worse execution price than expected), though it is more commonly associated with negative impacts on the trader’s strategy. Slippage is particularly significant in high-frequency trading, algorithmic strategies, and during volatile market conditions, where rapid price fluctuations make it difficult to execute trades at desired prices.
Liquidity is not static and can fluctuate based on market conditions, the actions of large market participants, and other phenomena endogenous to financial markets that we will explore more deeply further down. Various measures are used to quantify liquidity in the LOB, including bid-ask spreads, market depth, and the price impact21 Price impact in market microstructure literature refers to the apparent impact that an individual trade, or trades, have on the market price of the asset being traded. As we’ve already seen, and will continue to investigate, modeling price impact is highly non-trivial. of trades.22 Widely cited works in market microstructure research that discuss liquidity and its implications include Maureen O’Hara, Market Microstructure Theory (1995), which offers a comprehensive overview of how market structures impact trading and liquidity; and “Continuous Auctions and Insider Trading” by Albert S. Kyle (1985). Another important work is Jón Daníelsson and Richard Payne, “Measuring and Explaining Liquidity on an Electronic Limit Order Book: Evidence from Reuters D2000-2,” Bank of International Settlements website (https://www.bis.org) (3 October 2001), which explores the specifics of liquidity measurement in LOBs. Market microstructure research has developed several specific measurement statistics to quantify market liquidity, which are essential for understanding the ease with which trades can be executed without significantly impacting asset prices. It’s easy to get the terms mixed up, so let’s review them and trace their relationship to one another. Among the most commonly used measures is the bid-ask spread, which as we’ve seen represents the difference between the best available bid price (the highest price a buyer is willing to pay) and the best available ask price (the lowest price a seller is willing to accept). A narrower bid-ask spread indicates higher liquidity, as it suggests that buyers and sellers are willing to trade at prices close to each other, reflecting a more competitive market. Another critical measure is market depth, which you will recall assesses the volume of buy and sell orders available at different price levels in the LOB. Market depth indicates how much of an asset can be traded without causing significant price changes; greater market depth signifies higher liquidity, as large trades can be executed with minimal impact on prices. Depth is often analyzed at specific price intervals away from the best bid and ask, providing a more detailed view of how liquidity is distributed across the order book. Additionally, price impact measures the extent to which large trades affect the asset’s price. This impact is typically evaluated by observing the price movement resulting from a trade of a certain size, where a lower price impact implies higher liquidity, indicating that the market can absorb large orders with minimal price disruption. Price impact functions and Kyle’s lambda (introduced by Albert S. Kyle in 1985 – a model which we will explore more shortly) are tools used to quantify this effect, with Kyle’s lambda specifically measuring the linear relationship between trade size and price changes. Furthermore, the Amihud illiquidity measure, proposed by Yakov Amihud in 2002, captures the daily price impact relative to trading volume, calculated as the absolute price change per unit of trading volume, averaged over a period. A higher Amihud ratio indicates lower liquidity, as it suggests that price changes are more sensitive to trading volume. Lastly, the turnover ratio reflects trading activity in relation to the size of the market, calculated as the trading volume divided by the number of shares outstanding, where a higher turnover ratio suggests higher liquidity because it indicates that a large proportion of the asset is being traded frequently. These statistics offer various perspectives on liquidity, enabling researchers and market participants to assess the ease of trading and the resilience of the market to large orders.
Finally, let’s make sure to connect this concept of liquidity back to our knowledge of the limit order book. If you think back to the LOB examples, you will probably recall instances where a large market order came in and “chewed through” multiple price levels in the book, causing the market price to change. The term of art for this is walking the book – a concept long known intuitively to traders, but only recently examined empirically by market microstructure researchers. A LOB with high liquidity will be able to resist the ability of any one market order to walk the book, leading to fewer price changes; the bigger and more numerous the market orders that come in, the faster the LOB’s liquidity will be depleted. As you also saw in the examples, a liquidity refill refers to the process by which new limit orders may be added to the LOB, restoring liquidity at specific price levels. Just as new market orders potentially deplete liquidity and reduce the depth of the book, refills potentially restore liquidity and increase the depth of the book. Liquidity refills – whether done manually or with automated trading algorithms – stabilize the market by ensuring that there is always some level of liquidity available, which can reduce the volatility caused by large trades and maintain more orderly market conditions. It is this one-two operation of book-walking followed by liquidity refills (with eventual reversals in the directionality of the trades – from buy to sell, or vice versa) that constitutes a good deal of the liquidity shifts we see in LOBs.
The Players and Their Techniques
Having explored a simple model of price dynamics within the LOB, let’s now proceed to take a brief tour of the market participants who might make use of it. Who are they? What do they want? And how do they go about getting it? While not comprehensive, the discussion that follows will give you an overview of various important players, the moves they might make, and how these affect the order flow and hence the price determination of securities markets.
Traders
Let’s begin with the most simple sort of player: your typical trader. Only, no trader is really typical when you think about it. They might be a lone individual or the employee of a large firm – a retail trader looking to make money for themselves, or an institutional trader managing the assets of an investment bank, hedge fund, pension program, etc. It is very difficult to generalize about traders. They can adopt all sorts of trading strategies, from value investing that seeks to profit off the inherent value (whatever that is!) of presently undervalued securities to momentum investing that tries to buy securities that have already been moving up in price while selling off any securities that have been moving down in price over some recent period. They operate at different time scales, from day traders who buy and sell within a single trading day to position traders who hold onto securities long-term in the hopes they will continually appreciate. And indeed, different traders will play with different sorts of securities: some will deal in stocks, others in derivatives such as options and futures, yet others in the currency pairs of foreign exchange (forex) markets, and yet again others in the raw materials and agricultural products which the exchanges call commodities. There is much more to be said about the many different kinds of traders and their divergent perspectives than I can fit in this essay – stay tuned for future work on this subject – but for now it should suffice to say that the most common rule of thumb among them is one you’ve probably heard of: whatever security you deal in, buy it cheap and sell it dear, so that you can pocket the difference.
If you know anything about financial markets, you probably already had at least a vague sense of the above. But now that you’ve understood the basic mechanics of the limit order book, you should have a sense of how traders in general affect market prices and why that’s so important.
Because ultimately, it is the traders who are in the driver’s seat when it comes to the evolution of securities prices throughout the trading day, as we’ve already examined using simplified examples of LOBs. By submitting limit orders, traders set the prices at which orders may or may not get executed throughout the trading day. As these orders are continuously matched (or not) on the exchange’s order book, they drive price changes in real-time.
Market Makers
For a more complex example of a major player, we might look at market makers, one of the key providers of liquidity. These participants help ensure the stability of financial markets by continuously quoting buy (Bid) and sell (Ask) prices for various securities. They stand ready to buy or sell these securities from their own inventories, thereby facilitating trades and stabilizing prices through the liquidity this adds to the system.23 The role of market makers has evolved significantly from the early days of financial markets, where individual traders at exchanges specialized in providing liquidity for specific securities. These individuals, often known as specialists or floor traders, would stand ready to buy and sell particular stocks, ensuring that there was always someone willing to trade, thereby maintaining an orderly market. Over time, as markets grew in size and complexity, the role of market maker transitioned from one performed by individuals on the trading floor to one managed by large corporations equipped with the most advanced technology. These corporations now act as designated market makers (DMMs) for exchanges, managing liquidity across thousands of securities. Despite the shift from individual traders to sophisticated algorithms run by large firms, the basic business model of market making has remained largely the same: market makers earn profits by capturing the bid-ask spread. They also provide critical services to the market by ensuring continuous liquidity, facilitating smooth trading, and reducing price volatility. The evolution has been driven by the need for greater efficiency and the ability to handle larger volumes, but the fundamental principles of maintaining liquidity and earning profits from spreads have persisted throughout.
Examples of market makers include investment banks, brokerage firms, and specialized trading firms. For instance, firms like Goldman Sachs and Morgan Stanley act as market makers in equities, bonds, and derivatives, while proprietary trading firms such as Citadel Securities and Virtu Financial focus on high-frequency trading strategies to provide liquidity. Additionally, stock exchanges like the New York Stock Exchange (NYSE) designate certain firms as designated market makers (DMMs) to manage the trading of specific stocks, ensuring an orderly market. By bridging the gap between buyers and sellers, market makers play a vital role in enhancing market efficiency and price formation.
When the market makers update their bid and ask prices in the LOB, as they do continually based on market conditions and their own inventory levels, this is called a quote. These quotes facilitate liquidity and act as reference points in the market; they help to ensure that there are always buyers and sellers available as well as to maintain a stable and efficient trading environment.
Generally, market makers aim to maintain a neutral inventory position – that is, they want to begin and end the trading day with as close to 0 of their particular asset on their books as possible. So how, then, do these people make money? The market makers’ revenue structure is too complex to go into here, but an oversimplified version for our purposes is that they have two sources of income: (1.) they’re paid a commission by the exchange, for the service of providing liquidity; and (2.) they try to make money on the round trip trade of all the operations they undertake on a trading day, which basically amounts to whatever money they make on orders24 The most obvious way the market makers could make money is off their sell orders. However, buy orders aren’t just costs – they can make money, too, under certain conditions. For example, if the market maker is in a short position – if they’ve sold some quantity of a borrowed security – and the market price of that asset decreases, then when they buy as much of the asset as they’d sold to close the position they’ll have made more money overall (because they sold when it was high and bought when it was low). Market makers do this all the time since they want to end the day in a neutral inventory position. But any trader can make money the same way, if they think some asset will depreciate for whatever reason. Market makers are often juggling such typical trading considerations alongside their duty to provide liquidity to the market in general. They might do a short in order to pocket some money for themselves – or they might do it because they see an imbalance in the LOB and want to provide more liquidity on (say) the Ask side. minus their operating costs.
An aside on the transition from quote-driven to order-driven markets
The historical transition of financial markets from quote-driven to order-driven systems marks a significant evolution in trading mechanisms. In quote-driven systems, market makers or dealers play a central role by continuously providing buy and sell quotes for securities, thereby ensuring liquidity. These market makers profit from the spread between the bid and ask prices, and their willingness to trade at quoted prices stabilizes the market. This system was prevalent in many major exchanges, including the New York Stock Exchange (NYSE) and the London Stock Exchange (LSE), where human intermediaries facilitated most trades.
With advancements in technology and the rise of electronic trading, financial markets gradually transitioned to order-driven systems. In these markets, all participants can submit buy and sell orders directly into a central order book, where trades are matched based on price and time priority. This democratization of trading access led to increased transparency, as all market participants could view the same order book and the associated prices. The move to order-driven markets has reduced reliance on intermediaries, which has lowered transaction costs. Exchanges like NASDAQ and electronic communication networks (ECNs) spearheaded this transition, ultimately transforming the landscape of global financial trading.
While the shift to order-driven markets has transformed the landscape of financial trading, market makers continue to play a crucial role in providing quotes for securities prices, ensuring liquidity and stability. In contemporary order-driven markets, market makers are not just traditional intermediaries but sophisticated entities employing advanced algorithms to manage their inventory and mitigate risks. Their presence in the order book helps bridge gaps during periods of low trading activity, offering buy and sell quotes that enable other participants to execute trades efficiently. By continuously updating their quotes based on market conditions, they help maintain a balanced and orderly market. Thus, while the dynamics of their role have evolved, market makers remain vital to the smooth functioning of modern financial markets, complementing the decentralized nature of order-driven trading systems.25 The transition in securities markets from a quote-driven to an order-driven system parallels similar shifts in physical production industries, moving from a “push” to a “pull” model. In the former, firms would maintain large inventories and sell from stock, absorbing storage costs and restocking as needed. In the latter, technological advances enabled firms to scale production more efficiently, producing goods only as orders came in, thereby minimizing inventory. Likewise, in securities markets, the shift was facilitated by advancements in telecommunications and computing technologies, which enabled real-time tracking of orders and improved coordination between buyers and sellers. This allowed for more nimble price formation and liquidity provision without the need for market makers to hold large inventories of securities. For more on the transition in securities markets, see Hung-Neng Lai (2007). “The Market Quality of Dealer Versus Hybrid Markets: The Case of Moderately Liquid Securities,” Journal of Business Finance & Accounting 34, 349-373. For more on the transition from “push” to “pull” systems in industrial supply chains, see Shigeo Shingo, A Study of the Toyota Production System From an Industrial Engineering Viewpoint (1989), pp. 97-119.
The Exchanges
As I’ve discussed at length, it is market participants themselves who are in charge when it comes to setting prices on securities markets. Stock exchanges, such as the New York Stock Exchange (NYSE), play a supportive role in the price-administration process by providing the algorithmic procedure by which market orders are matched to limit orders to execute a trade. Beyond this, prices are for the most part administered by traders in the pursuit of their many and sundry strategies.
That said, exchanges do in certain very narrow and particular contexts take a more direct role in administering at least some securities prices – particularly during the opening and closing auctions, where they determine the reference prices for the beginning and end of the trading day. These reference prices are calculated based on the aggregation and matching of orders submitted by market participants. The exchanges have the authority to intervene in this process to ensure orderly markets, especially during times of significant imbalance, by facilitating additional liquidity or adjusting the mechanics of the auction. However, the primary function of the exchange even in these moments is to act as a facilitator that organizes and reflects the intentions of the market participants, rather than independently determining prices per se.
It’s important not to exaggerate the role of the auction as a determining factor, however. The opening and closing call auctions on stock exchanges, like those at the NYSE, differ significantly from the auctions theorized by early economists such as Léon Walras (whose work we shall turn towards in more detail further down), particularly in their execution and the role of price formation. Walrasian auctions are theoretical constructs where prices adjust continuously until supply equals demand, resulting in a market-clearing equilibrium. In contrast, stock exchange call auctions are discrete events where orders are collected and matched at specific times (opening and closing), and the final price is determined by the highest volume of matched orders, rather than a continuous adjustment process. Moreover, stock exchanges allow for interventions by market makers to manage imbalances, whereas Walrasian auctions assume no such external influence. And, as of course we’ve seen, during the rest of the trading day there is no discernible equilibrating mechanism in anything that happens in the limit order book, and little reason to believe such a mechanism exists.
To put the call auctions into terms of price administration, some market participants at the beginning and end of the trading day are essentially telling the exchanges that they are okay with ceding their price-setting to the exchange (and where applicable, designated market makers assisting exchanges). Similar to how those submitting market orders during continuous trading periods are essentially telling the LOB, “I’m ceding my ability to price-set to others in exchange for certainty of execution,” participants in the opening and closing call auctions are ceding their ability to price-set to the exchanges (who are arguably also “participants” in this instance – albeit a special class of them who are providing everyone with the trading venue themselves.)
The reference prices that exchanges arrive at during their call opening and closing auctions are indeed functions of the priced orders previously submitted by market participants. In these call auctions, all orders for each particular security are gathered and matched at a single price that maximizes the volume of trades. This price is determined by the limit orders (i.e., priced orders) that participants have already submitted. The exchange’s algorithm considers these orders to find the price where the highest number of shares can be traded, often considering factors like minimizing order imbalance and maximizing the traded quantity.
The table below summarizes six of the largest exchanges and the basic schematics of a trading day at each, along with their chosen matching algorithm for the continuous trading period.
| Stock Exchange | Matching Engine Rule | Opening Auction | Closing Auction | Operating Hours (Local Time) |
|---|---|---|---|---|
| New York Stock Exchange (NYSE) | Hybrid Rule: Primarily Price-Time Priority, but with NYSE DMM (Designated Market Maker) intervention | Opening auction (Call auction) | Closing auction (Closing Cross) | 9:30 AM – 4:00 PM (ET) |
| NASDAQ | Hybrid Rule: Price-Time Priority with Order Imbalance Indicator used for price “discovery” | Yes, opening cross (NASDAQ Opening Cross) | Yes, closing cross (NASDAQ Closing Cross) | 9:30 AM – 4:00 PM (ET) |
| Tokyo Stock Exchange (TSE) | Price-Time Priority | Yes, opening auction (Itayose method)26 The Itayose method for determining the closing price is very similar (all of the closing auction methods are much more similar to each other than they are different) to the other methods, though there are subtle differences: it involves several conditions to ensure an orderly auction. First, the closing price is chosen where bids and offers match within one tick above the highest order price and one tick below the lowest order price. If multiple prices meet this condition, the price that maximizes traded volume is selected. If a tie remains, the price that minimizes the difference between the cumulative volume of sell and buy orders (the “surplus volume”) is chosen. If there is still a tie, the lowest price is selected if the cumulative sell volume exceeds the buy volume, or the highest price if the buy volume exceeds the sell volume. Finally, if imbalances exist, the price is selected based on how it compares to the Reference Price, favoring the price closest to it within the range where the imbalance is smallest. | Yes, closing auction (Itayose method) | 9:00 AM – 11:30 AM, 12:30 PM – 3:00 PM (JST) |
| Shanghai Stock Exchange (SSE) | Price-Time Priority | Yes, opening auction (Call auction) | Yes, closing auction (Call auction) | 9:30 AM – 11:30 AM, 1:00 PM – 3:00 PM (CST) |
| Hong Kong Stock Exchange (HKEX) | Price-Time Priority | Yes, opening auction (Pre-opening session) | Yes, closing auction (Closing auction) | 9:30 AM – 12:00 PM, 1:00 PM – 4:00 PM (HKT) |
| Euronext | Price-Time Priority | Yes, opening auction (Call auction) | Yes, closing auction (Call auction) | 9:00 AM – 5:30 PM (CET) |
IPOs
Let’s go back to thinking about stocks, the most famous kind of security. So traders set prices as market orders are matched to limit orders; exchanges set reference prices at the start and end of the trading day to maximize executed orders at that moment, based on previous orders placed by participants in the limit order book; and at any given instant, the market price of the stock is nothing more than the price of the last executed order. That accounts for stock prices most of the time. But you may find yourself asking: wait a second, companies only start selling stocks on financial markets when they go public, which is a long and drawn-out process. Who sets the initial price of a company’s stock – known as its initial public offering (IPO) – before it becomes publicly listed on an exchange, and how? On what basis? To whose benefit?
Although a complete analysis of the IPO process is beyond the scope of this essay, for the sake of completeness, allow me to give you a general description of the IPO price-setting procedure. Here, too, we can examine the process concretely using a price-administrative framework.
Determining the IPO price for a company going public is a multi-step process involving both the company and corporate underwriters who typically work for investment banks. The process begins with the underwriters conducting a thorough valuation of the company, which includes analyzing financial statements, assessing market conditions, and comparing the company to similar firms that are already publicly traded. One common method used is the discounted cash flow (DCF) analysis, where future cash flows of the company are estimated and then discounted to their present value to determine an intrinsic value. Another method is the comparable company analysis (CCA), where the company’s financial metrics, like earnings and revenue, are compared to those of similar publicly traded companies to establish a valuation range.27 Yadav, Ajay, Jaya Mamta Prosad, and Sumanjeet Singh. 2023. “Pre-IPO Financial Performance and Offer Price Estimation: Evidence from India” Journal of Risk and Financial Management 16, no. 2: 135. https://doi.org/10.3390/jrfm16020135
Once the valuation is established, the underwriters gauge investor interest through a process called book building.28 Once the order book is fully built, and the underwriters have a clear picture of demand at various price points, they set the final IPO price. This price reflects a price level that satisfies both the company’s capital-raising objectives and the investors’ expectations for future returns. The final step involves allocating shares to investors, often prioritizing long-term institutional investors who are more likely to hold the stock rather than flip it for short-term gains. In market microstructure terms, book building is akin to a continuous auction process where underwriters act are essentially acting as market makers in the primary market for stocks (as opposed to in the more familiar (and much larger) secondary market of the exchanges), matching buyers and sellers while dynamically adjusting the auction parameters (i.e., price and quantity) based on real-time feedback from investors during the roadshow. During this phase, they conduct what are referred to as roadshows, where the company’s management presents its business case to potential investors, allowing the underwriters to collect feedback on the price range they would be willing to pay. Based on this feedback, the underwriters and the company narrow down the IPO price range.
Finally, after considering factors like market demand, overall market conditions, and the company’s financial needs, the underwriters set the final IPO price. This price aims to balance the company’s desire to raise capital with the need to ensure strong post-IPO performance by leaving some potential upside for new investors. For instance, if investor demand is high, the final price may be set at the upper end of the range or even above it. Conversely, if demand is lukewarm, the price might be set at the lower end or even below the initially proposed range.
Metaorders
Let’s return once again to our main story of price formation in limit order books. Having established a foundational understanding of market microstructure, including the intricacies of LOBs and the role of market makers, we can now look into a somewhat more advanced (though no less foundational, as I will argue) topic: metaorders and the traders who place them.
Metaorders are large trading orders that are strategically executed over a period of time, rather than all at once, to minimize market impact and achieve a more favorable average execution price.29 Viktor Bazylevych and Vitalii Ihnatiuk (2019). “Metaorder limit prices in evaluating expected market impact and assessing execution service quality.” Investment Management and Financial Innovations, 16(2), 355-369. doi:10.21511/imfi.16(2).2019.30 These orders are broken down into smaller, manageable portions to avoid sudden price movements that could result from a single, large trade walking the book – or from other traders noticing the large order and revising their orders in light of perceived urgency of whoever submitted the large order.
Typically employed by institutional traders, metaorders require sophisticated algorithms and detailed market analysis to optimize execution. By carefully timing and placing these smaller orders, traders can obscure their true intentions from the market, thereby reducing the risk of adverse price changes that could be triggered by revealing the full size of their trading position. This is a very common form of strategic behavior in the LOB.
As the metaorder is broken down into smaller portions, these individual trades gradually consume the liquidity available at various price levels in the order book. The hope of the trader who adopts a metaorder strategy is that this effect will happen slowly enough, and in small enough portions relative to the liquidity refills to the LOB, that the price will stay the same for the entire order, rather than changing adversely (buying becoming more expensive, selling less profitable) as the order is executed.30 Vincent Van Kervel & Albert Menkveld, “High-Frequency Trading around Large Institutional Orders,” The Journal of Finance 74:3 (June 2019), pp. 1091-1137.
Now, there’s no reason to believe they’ll always succeed, of course. A metaorder can still significantly impact the price within the LOB – albeit spread out over a longer time period – due to its large size; and this will especially be the case if the metaorder execution strategy were to be prematurely discovered.
If a substantial portion of the metaorder matches with existing limit orders, it can contribute toward a depletion of the best Bid or Ask prices, causing the market price to change. For example, a large buy metaorder can cause the market price to move upward as it absorbs the available priced sell orders, beginning with the best Ask, and continuing down the book. Conversely, a large sell metaorder can drive prices down by flooding walking down the price levels of the buy side of the book. A LOB without sufficient depth may find itself giving way before the metaorder despite its piecemeal arrival.
Furthermore, when not properly monitored by the trader who is conducting it, a metaorder’s gradual but persistent execution can still create trends in the price movement, influencing other traders’ perceptions and actions within the market. As the metaorder slowly hits the LOB, observant traders may detect the unusual patterns of consistent buying or selling. This can lead them to infer the existence of a large underlying order, prompting strategic responses. Some traders might attempt to front-run the metaorder by placing their orders ahead of the anticipated price movement, to capitalize on the expected trend. Others might withdraw their orders, anticipating adverse price changes, thus reducing liquidity and exacerbating the metaorder’s impact. The perceived order imbalance can create a feedback loop, where the market reacts not only to the actual trades currently underway but also to expectations about the behaviors of other participants, further amplifying price movements.
The concept of the metaorder first emerged formalistically in the theoretical discussions of market microstructure in the late 70s, though an intuition of its existence among traders long predates its inclusion in modeling efforts., Theory became more sophisticated as financial markets became more computerized. Early studies by economists and financial theorists began to explore the impact of large orders on market prices and liquidity. Researchers postulated that institutional investors, who typically handle significant volumes of trades, would break down their large orders into smaller ones to mitigate market impact and avoid signaling their trading intentions to the market. These early theories were grounded in the observation that executing a large order in one go could cause substantial price disruptions, making it less favorable for traders seeking optimal execution prices.
The practical discovery and empirical validation of metaorders came with the advent of advanced trading technologies and the increased availability of high-frequency trading data in the late 90s and early 2000s. As algorithmic trading became more prevalent, it became possible to analyze and identify patterns consistent with the strategic execution of large orders. Researchers and market analysts began to uncover the sophisticated techniques used by institutional traders to distribute their large orders over time, confirming the earlier theoretical predictions that were based on comparatively more sparse empirical work. This period also saw the development of algorithms specifically designed for executing metaorders, such as VWAP (Volume Weighted Average Price) and TWAP (Time Weighted Average Price). (We will explore other more bespoke execution algorithms later on.) The ability to recognize and model the impact of these orders has since become a crucial aspect of market analysis and strategy development for both institutional and individual traders.
The Upshot
If you’ve made it this far, congratulations! You’re already ahead of the curve compared to most economists and commentators in understanding the micro-level dynamics of LOBs, and by extension, some of the basic dynamics of security price administration.
As I hinted at earlier towards the start of the essay, the question of who sets security prices (and how) is, on one level, straightforward: traders set the market price directly by submitting limit orders – priced orders – into LOBs, thereby providing the liquidity necessary to ensure a resilient order flow.
But there is another, more complex side to the story. We’ve gone through what is happening at a concrete, simplified level. Now we must turn to the question how price formation perpetuates itself. We need to learn more about the resilience of order flow in the face of substantial fluctuations in trading activity, and rapid shifts in available liquidity. We also need to begin learning about past efforts to model these complex dynamics.
The LOB serves as the central mechanism through which liquidity is managed and prices are formed for a given security in modern financial markets. By analyzing its basic operations and dynamics, we gain a clearer understanding of how orders interact to determine market prices, how liquidity is both provided and consumed, and the critical role that limit orders play in maintaining an orderly market. Liquidity, as defined in market microstructure, is not just the availability of buyers and sellers, but the depth, resiliency, and tightness of the LOB, which together influence the ease and cost of trading. This foundational understanding of the limit order book sets the stage for a deeper exploration of market microstructure theory.
I will now trace the intellectual history that has shaped our best current views on how markets function, how prices are formed, and the ongoing evolution of trading mechanisms and market participants. As we transition into this next section, we will dive into the key theories and models that have been developed to explain the complexities of financial markets, starting with the seminal ideas that have driven market microstructure research.
Part II:
An Intellectual History of Market Microstructure
Having established an understanding of the mechanisms of LOBs, the role of market makers, and the complexities of metaorders—we now turn our attention to the intellectual history of market microstructure theory. This field, which emerged formally in the latter half of the 20th century, has provided insights into the micro-level workings of financial markets, particularly regarding how prices are formed and the dynamics of trading. By examining the evolution of market microstructure theory, we can appreciate the foundational ideas that underpin modern trading practices and the models used by market participants today.
A key focus within this intellectual journey has been the theorization of metaorders and their impact on price. Early theorists sought to understand how large, strategically executed orders influence market prices and liquidity. Over time, their models have evolved to capture the nuanced behaviors of institutional traders and the resulting market effects. This section will explore the intellectual contributions of a range of thinkers from varying theoretical (and ideological) frameworks.
Early Models
Léon Walras (1874)
The so-called Walrasian Auctioneer31See Bouchaud et. al., p. 6-9 model is a theoretical construct, first proposed by Léon Walras in the late 19th century, which describes an idealized market mechanism for setting prices in financial markets. Under this model, an auctioneer hypothetically calls out prices for goods and services, and then buyers and sellers submit quantities demanded and supplied at those prices. There is a strict functional relationship between the auctioneer’s price and the buyers’ and sellers’ quantities; when the former changes, so too do the latter in exact proportion. The auctioneer adjusts the prices iteratively, causing the quantities supplied and demanded to adjust in turn, until a market equilibrium is reached where the total quantity demanded equals the total quantity supplied. This process ensures efficiency in the allocation of resources and that the market can clear without excess supply or demand.
One of the strong points of the Walrasian Auctioneer model is that it has a very straightforward mathematical formulation, which allows for the derivation of equilibrium prices and quantities under the assumption of perfect competition. It provides a basic building block for general equilibrium theory by showing how decentralized markets can yield fundamentally efficient outcomes. The model assumes – somewhat breathtakingly, this author thinks – that all participants have perfect information and that there are no transaction costs or frictions. This simplifies the analysis and makes the model a powerful tool for theoretical exploration, though not particularly useful for real-world modeling.
The Walrasian Auctioneer model has several notable shortcomings. First, the assumption of an auctioneer who continuously adjusts prices until equilibrium is reached is unrealistic, as no such central figure exists in real markets. Presumably this is a metaphor for some emergent process that markets undertake in a more piecemeal and decentralized fashion, but if you try to observe it happening in the wild it will prove frustratingly elusive.
Additionally, the model assumes that all trades occur only at equilibrium prices, ignoring the dynamics of real-world trading processes where prices are constantly in flux due to any number of potential issues – several of which we’ve already explored, and a few more that we shall visit further along (information asymmetry, market impact, order flow, liquidity constraints, and strategic behavior by market participants all being potential candidates).
Consequently, however important the Walrasian Auctioneer model was to the development of neoclassical models, it falls short in providing a comprehensive understanding of the actual mechanisms, strategic behaviors, and really-existing decentralized price-administration that is driving market behavior, and with it, price formation.
To give you an idea of the absurdity (sorry) of Walras’ idea for the general equilibrium, consider the following brief worked numerical example. Suppose we have a simple market for apples. There are three buyers and three sellers. Let’s say we have a market for apples, where multiple buyers and sellers exist. The auctioneer’s job is to find the equilibrium price where the quantity of apples demanded by buyers equals the quantity supplied by sellers:
Initial Prices and Quantities
- Initial Price: The auctioneer starts by announcing an initial price of $2 per apple.
- Quantity Demanded (at $2): At this price, buyers are willing to purchase 50 apples.
- Quantity Supplied (at $2): Sellers are willing to sell 70 apples at this price.
Check for Equilibrium
- Excess Supply: At the price of $2, there is an excess supply of 20 apples (70 supplied – 50 demanded).
- The auctioneer observes that the supply exceeds demand and understands that the price is too high.
Adjusting the Price
- Lowering the Price: To reduce the excess supply, the auctioneer lowers the price to $1.50 per apple.
- Quantity Demanded (at $1.50): At the new price, buyers now want to buy 60 apples.
- Quantity Supplied (at $1.50): Sellers are now willing to sell 60 apples.
Rechecking for Equilibrium
- Supply Equals Demand: At the price of $1.50, the quantity demanded equals the quantity supplied (60 apples each).
- Since there is no excess supply or demand, the auctioneer has found the equilibrium price.
Concluding the Auction
- The auctioneer announces that the equilibrium price is $1.50 per apple.
- At this price, the market clears, meaning that all apples supplied by sellers are purchased by buyers, and no excess supply or demand remains.
I’d like to return briefly to the discussion of closing call auctions from earlier. In real-life call auctions, we see auction designs that permit behavior that would not be allowed under Walrasian assumptions. The original Walrasian auctioneer model is inapplicable to existing call auctions for three core reasons, according to the logic of their own model: not everyone participates in the closing auction; those who do participate can see each other’s orders and have the opportunity to revise or cancel them in response, behaving strategically right up until the auction’s end before a reference price is obtained and executed against the highest possible volume; and most closing call auctions still end with imbalances, meaning they don’t clear. The original Walrasian Auctioneer model doesn’t account for these factors.
If proponents of the Walrasian model want to redefine what constitutes market clearing or equilibrium price, that’s fine, and some have seen fit to do so; but as we’ll see, this train of thought sooner or later diverges considerably from what Walras originally intended, making it a less-than-meaningful exercise. It would be more useful for a theorist to read the actual closing auction manual for the NYSE and interpret it in terms of order flow through the closing auction LOB, just as the more sophisticated traders involved in the closing auction would.
Walras’s true significance lies in being the starting point for everyone in the field. But this poses a challenge for the theory of market microstructure, as that theory only became realistic to the extent that it split off from Walras’s models. Initially, researchers tried to improve his framework by adding exceptions and nuances to make things more realistic. However, these exceptions soon proved to be more important than the original theory itself, leading to the development of increasingly comprehensive alternative models. While these new theories still paid lip service to equilibrium models, they gradually became less connected to them. Look closely enough at them, and you’ll realize they teach us there’s no need for an equilibrium concept whatsoever. Thus, we must begin with Walras, but everything that follows will progressively move further away from his ideas. And this departure from neoclassical economics marks the true beginning of the science of market microstructure.
Bachelier (1900)
Louis Bachelier’s model for changes in stock prices, as formulated in his 1900 dissertation “Théorie de la Spéculation,” can be regarded as one of the pioneering works in financial mathematics. Bachelier was of the view that the prices of stocks follow a stochastic process that can be approximated to Brownian motion,32 Originally observed as the random movement of particles suspended in a fluid, Brownian motion (named after the botanist Robert Brown, who first observed it in 1827 while looking at pollen particles under a microscope) was mathematically formalized by Albert Einstein and Norbert Wiener. It describes a continuous stochastic process where the particles’ changes in position are normally distributed and independent over time. In financial markets, Einstein and Wiener’s formulas are applied to price changes of stocks and other financial instruments, underpinning models like the Black-Scholes option pricing framework. The primary appeal of Brownian motion to financial analysts lies in its ability to capture the randomness and unpredictability of market movements. However, as discussed above, the model’s theoretical shortcomings include its assumption of normally distributed returns, which fails to account for the observed fat tails and skewness in real market returns, and its assumption of constant volatility, which does not reflect the empirical reality of volatility clustering. On a deeper theoretical level, models like Black-Scholes tend to downplay the critical role of agency and the complex interdependence among market participants, who are not mere particles but players in a dynamic feedback loop. This concept is explored in frameworks like George Soros’s theory of reflexivity, Frederic S. Lee’s theory of structured agency, and Marx’s theory of dialectical materialism, all of which emphasize that participants and their environment continually shape each other. Models that fail to account for this intricate path of causation between participants and the market as a whole are inherently incomplete. a physical process characterized by continuous and has random movements. Bachelier assumed that price changes are normally distributed and independent across time, which means that past movements of prices do not influence future movements. This was a radical departure from the deterministic models of the time and laid the foundation, for better or worse, for the concept of market randomness, a key premise of numerous later models of market impact.33 Bachelier, L. (1900) Théorie de la Spéculation. Gauthier-Villars, Paris. Bachelier was one of Henri Poincaré’s students at University of Paris.
Another important contribution of Bachelier’s model was the notion of “fair game” or “martingale.” If the efficient-market hypothesis34 The efficient-market hypothesis asserts that asset prices fully reflect all available information, and that a major consequence of this, if correct, is that one should not see recurrent structure or patterns to prices or price returns that would be exploitable by traders. holds, then the expected value of the future price of a stock is its current price. That is, the expected gains due to changes in price are balanced by the risks, which is a manifestation of the absence of arbitrage opportunities in an efficient-market. Bachelier’s mathematical approach employed the use of probability theory in an attempt to derive both the expected value and variance of stock prices.
The model was, for its time, revolutionary, though it did have its failings, particularly in its assumption that changes in price were normally distributed, which bears little resemblance to the fat tails and asymmetry empirically observed in financial markets. Moreover, Bachelier’s insistence on the randomness of these price moves does not conform to reality. The stock market – famously not quite random35 Lo, Andrew W., and A. Craig MacKinlay. “Stock Market Prices Do Not Follow Random Walks: Evidence from a Simple Specification Test.” The Review of Financial Studies 1, no. 1 (1988): 41–66 – would certainly be much simpler to model if it were! Instead, as we’ll see later on, real-world order flow is quite complex, and consequently, so, too, is the task of modeling it. Even our simplified numerical examples from earlier show that the arrival of new orders can quickly alter the rest of the orders in the LOB – and with them, the potential for price changes.
Nonetheless, Bachelier’s contributions laid the groundwork for later advancements, influencing the Black-Scholes option36 We will discuss options, what they are, and the Black-Scholes option pricing model in depth later on. pricing model and the development of stochastic calculus, solidifying his place as a foundational figure in the field of quantitative finance.
Market microstructure models from the middle period of the field’s development (the ’70s through the ’80s) can generally be grouped into two categories: inventory models and information-based models.
Inventory-Based Models
Garman (1976)
The Garman model of dealer capital and inventory37 Garman, M. B. 1976. “Market Microstructure.” Journal of Financial Economics, 3(3), 257-275. is a theoretical framework developed by the economist Mark Garman to explain how market makers (dealers) manage their inventory of securities and how this management influences their pricing strategies, ultimately determining overall market liquidity. Central to the model is the concept that dealers adjust their prices based not only on order flow, but also on their current inventory levels, in order to mitigate risks associated with price fluctuations. The model posits that dealers aim to maintain an optimal inventory level by setting prices that encourage trading in a direction that corrects any inventory imbalances. High inventory levels might lead dealers to lower prices to stimulate sales, whereas low levels could drive prices up to curb buying. This inventory-based pricing strategy is crucial for understanding how dealers contribute to market liquidity and stability, and it provides insights into the dynamics of bid-ask spreads and trading volumes in financial markets.
Dealers maintain inventories of securities and are thus exposed to price-change risks. The model thus posits an idea of inventory risk management, according to which securities are priced so as to favor returning to some target or optimal level of inventory. The prices set by a dealer, consequently, will reflect current conditions of inventory. The pricing strategy has a component that rewards the risk of carrying inventory. Dealers want to minimize their cost of inventory holding while balancing it against the need to provide liquidity to the market. This trade-off impacts their capital requirements and the amount of risk they wish to incur. Indirectly, through price adjustments in response to variations in levels of inventory, market liquidity is controlled by the dealers. Greater inventory risk could be associated with wider bid-ask spreads that reflect the greater compensation for the risk taken by the dealer.
Information-based Models
The Kyle Model (1985)
Albert S. Kyle’s model is influential in the economics of finance for its understanding of the dynamics of market liquidity, information, and pricing. The Kyle Model essentially describes how informed traders (or insiders) and noise traders impact the market.38 Kyle, Albert S.1985. “Continuous Auctions and Insider Trading.” Econometrica 53, no. 6: 1315–35. https://doi.org/10.2307/1913210. It assumes three types of participants: informed traders, who possess private information about the fundamental value of an asset; noise traders (or liquidity traders), who trade on reasons unrelated to the asset’s fundamental value; and market makers, who set prices not according to some fundamental value, but rather by systematically adjusting the shape of the LOB, submitting limit orders on both sides of the book.
M. In this model, informed traders use their private information to make profits, while market makers adjust the prices in response to the order flow, trying to infer the information possessed by the informed traders based on the trades being made. Noise traders provide a cover for informed traders, making it difficult for market makers to distinguish between trades based on private information and those made for other reasons.
Thus, we find that the informed trader’s optimal strategy is a linear function of the deviation of the asset’s true value from its mean, and the market maker sets prices based on a linear function of the total order flow, reflecting both informed and noise trader impacts. The model elegantly captures the interplay between asymmetric information and market pricing.
The Kyle model is not without its theoretical shortcomings, however. One of its most notable limitations is its prediction that the arrival of a metaorder would result in a linear price impact. According to Kyle, as a metaorder is executed, the price moves in a straightforward, proportional manner with the size of the order. However, empirical evidence suggests a different reality: the price impact tends to follow a square root function.39 Quite a bit more on this point later. This discrepancy highlights a fundamental flaw in the Kyle model, as it oversimplifies the complex dynamics of market behavior. The actual square-root price impact reflects a more nuanced interaction between order flow and liquidity, suggesting that as order size increases, the marginal impact on price diminishes. This deviation from linearity underscores the need for models that better capture the intricate, nonlinear nature of real-world trading.
Glosten-Milgrom Model (1985)
The Glosten-Milgrom model40 Lawrence R. Glosten, Paul R. Milgrom,1985. “Bid, ask and transaction prices in a specialist market with heterogeneously informed traders”, Journal of Financial Economics, Volume 14, Issue 1., Pages 71-100, ISSN 0304-405X, https://doi.org/10.1016/0304-405X(85)90044-3. is a framework in market microstructure theory that describes how prices are set in the presence of asymmetric information. The model postulates that some traders possess private information about the true value of a security, while others trade based on public information or for liquidity needs.
This asymmetry in information affects market makers’ behavior when they have to quote bid and ask prices without having any idea whether the other party to the transaction is an informed trader or not. Market makers’ strategy would be to spread a compensation mechanism for potential adverse selection – a loss they might get after trading with informed traders. The spread gives them a reasonable possibility that their gains after some time from trading with the uninformed traders will offset the losses from trading with the informed ones.
The model continues by describing how market makers would have to alter their prices after the completion of some number of trades. When either a buy or a sell order comes in, the price of the transaction reflects the market maker’s revised opinions about the asset’s true value. This process of revision ought to reflect the probabilistic evaluation of the likelihood that the trade was an endogenously generated trade by an informed trader. The Glosten-Milgrom model consequently shows how prices approach the true value of the asset as trading activity reveals new information. This self-sustaining process of adjustment of prices toward equilibrium indicates a lively method of price formation in markets with asymmetric information.
One significant limitation of the Glosten-Milgrom model is the assumption that market makers can fully transact their inventory by the end of each day, a simplification it shares with the Kyle model.41 Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, Quotes and Prices. 2018. Cambridge University Press, p. 308. This assumption overlooks the complexities of inventory management and the risks associated with holding positions overnight. In reality, market makers often face constraints related to capital, risk limits, and the costs of carrying inventory, which can impact their pricing and trading strategies. Additionally, the model assumes a simplistic structure of informed and uninformed traders, without accounting for the diverse motivations and behaviors of market participants. These limitations suggest that while the Glosten-Milgrom model provides a foundational understanding of price setting in the presence of asymmetric information, it requires refinement to more accurately reflect the intricacies of real-world financial markets.
In concluding our examination of the Kyle and Glosten-Milgrom models, it is evident that both frameworks share significant similarities in their approach to understanding price formation in markets characterized by asymmetric information.
Both models emphasize the central role of informed traders and the adjustments made by market makers in response to the potential presence of these traders. They illustrate how the bid-ask spread functions as a crucial mechanism for managing the risk associated with trading against informed participants. Furthermore, both models incorporate the assumption that market makers can fully transact their inventory by the end of each trading period, a simplification that highlights their theoretical nature while also pointing to a common limitation in capturing the complexities of real-world market dynamics. These shared features underscore the foundational contributions of the Kyle and Glosten-Milgrom models in advancing our understanding of market microstructure, while also paving the way for further refinements and the development of more comprehensive theories grounded in emerging empirical evidence.
As influential as these earlier models were to the field of market microstructure and financial economics more broadly, what they lacked fundamentally were robust transaction-level microstructural datasets from which to glean characteristic empirical properties of order flow. This would change years later, when a group of economists interested in the empirics decided to put down some capital of their own to find out.
While earlier models of market microstructure, such as the Walrasian auctioneer model, the Bachelier Model, the Garman Model, the Kyle Model, and the Glosten-Milgrom Model, have significantly advanced our understanding of how markets operate, they often relied on certain ad hoc assumptions about market behavior. These models collectively provided valuable insights into price formation, liquidity, and the role of information in financial markets, yet they were limited by their reliance on simplified representations of market participants and their interactions. The assumptions underpinning these models, though useful for theoretical exploration, sometimes strayed from the complexities observed in real-world markets. In contrast, contemporary market microstructure research, led by what I term the “order flow theorists,” has shifted towards a more rigorous empirical approach. These researchers have distinguished themselves by focusing intensely on the detailed, data-driven analysis of order flow and trading behavior, aiming to ground their models in observable market phenomena and to provide a more accurate and nuanced understanding of how prices evolve in modern financial markets.
Part III:
Introducing the Order Flow Theorists
In the course of traversing the historical landscape of market microstructure models, we’ve explored the intricacies of market maker inventory-based pricing through the Garman model and dissected the Kyle model’s approach to price impact. While these seminal theories have laid the groundwork for our understanding of how market makers and large trades influence price dynamics, they are not without their limitations. The Garman model, with its focus on inventory adjustments, and the Kyle model, with its linear price impact predictions, both fall short of fully capturing the complex, nonlinear realities of modern trading environments. As we turn the page on these foundational theories, we recognize the need for a more comprehensive framework that reflects the nuanced interplay of trading activities in today’s financial markets.
Enter order flow theory, a modern paradigm that shifts our focus sharply further into the empirical dynamics of financial markets. Unlike its predecessors, order flow theory leverages comparatively vast historical datasets from high-frequency trading to examine the granular details of how orders are placed, matched, and executed within the market. It considers the full breadth of participants’ interactions and the resulting patterns of order flow that drive price formation. By examining the state of the LOB and the nonlinear impact of trades, this theory provides a more robust understanding of the mechanisms at play in financial markets. We will explore how order flow theory not only addresses the shortcomings of earlier models but also offers fresh insights into the real-world behavior of financial markets, illuminating the path toward a more nuanced and accurate depiction of price setting in contemporary trading.
Market Microstructure’s Copernican Revolution
In the late 1990s to early 2000s, several pioneering projects by academic researchers aimed to acquire and format LOB data, laying the groundwork for more sophisticated analyses of market microstructure. Although computerized transaction-level data from trading venues had been sporadically available since the late 70s, it was not until this pivotal time period at the turn of the century that quantitative researchers gained access to datasets of sufficiently high volumes and quality to generate new models.
One of the earliest and most influential datasets came from Island ECN (“Electronic Communication Network” – a type of electronic trading venue), which was later acquired by Nasdaq and was incorporated into the trading platform Inet.42 Another ECN, this one developed by Instinet, one of the largest equity trading and execution services firms in Europe. Researchers such as Terrence Hendershott,43 Barclay, Michael J. and Hendershott, Terrence J. and McCormick, Tim, “Information and Trading on Electronic Communications Networks” (February 18, 2002). Charles M. Jones, and Albert J. Menkveld utilized this data to study high-frequency trading, liquidity, and market efficiency.44 There were several important efforts going on largely concurrently at the time. For instance, the Toronto Stock Exchange was another early provider of LOB data. Researchers Thierry Foucault, Ohad Kadan, and Eugene Kandel used TSX data to examine the behavior of limit order traders and the impact of order flow on prices. In 2002, the NYSE OpenBook provided real-time access to LOB data. Researchers like Joel Hasbrouck and Gideon Saar used this data to study market transparency, order flow, and liquidity provision. Later that same year, Nasdaq TotalView began to offer comprehensive order book data beyond the top of the book, including all displayed quotes and orders at every price level. Providing access to LOB data became a significant source of revenue for many exchanges, and later financial data companies like FactSet would emerge specializing in providing these and myriad other financial datasets and analysis tools.
These rapid increases in the availability of data didn’t stop after the 2000s, either. Another highly important database to emerge out of this trend was the cutely named LOBSTER database,45 LOBSTER. “All About LOBSTER and High-Frequency Data.” Accessed October 29, 2024. https://lobsterdata.com/. developed by a team of financial econometricians at Humboldt University and the University of Vienna in the early 2010s. LOBSTER is a comprehensive, continuously updated database of trade-level LOB data, meticulously annotated and made available to application developers, who offer apps to HFTs and other traders as a product line. Independent, trade-level data warehouses like LOBSTER (now maintained by a company started by the original research group, called Frischedaten UG) compete with exchanges to offer robust datasets for use in the training of machine-learning algorithms and other trading software deployed by HFTs and market makers. These data are the lifeblood of any quantitative trading firm, offering unparalleled potential insights into order flow.
LOBSTER generates two files for each active trading day of a selected ticker: a ”message” file and an ”orderbook” file. The ”orderbook” file captures the evolution of the limit order book up to the specified number of levels, while the ”message” file records the events causing updates to the order book, with timestamps accurate to at least milliseconds and up to nanoseconds. These files, in .CSV format, can be easily read by statistical software. The ”message” file is organized as an Nx6 matrix, where each row represents an event such as a limit order submission, cancellation, deletion, or execution. Each event entry includes a unique order ID, size, price, and direction, detailing how it impacts the limit order book. The corresponding ”orderbook” file shows the state of the limit order book after each event. If the requested number of levels exceeds the available levels, dummy values are used to fill the gaps, ensuring a consistent output.
Special handling is applied during trading halts, with type ”7” messages in the ”message” file marking the halt and resumption, and the ”orderbook” file duplicating the preceding state during the halt. The historical data provided by LOBSTER covers all NASDAQ trading days from January 6th, 2009, to the day before yesterday, with plans to extend further into the past. Events that do not affect the limit order book, such as net order imbalance indicators, are skipped by the limit order book reconstruction algorithm. This ensures that only relevant updates are reflected in the output, keeping the data concise and focused on the evolution of the limit order book.
These data have ended up serving as the basis for studies further confirming some startling statistical regularities embedded within otherwise turbulent financial markets.
The Empirical Properties of Order Flow
Sign Autocorrelation
Long-range autocorrelation of order signs in market microstructure refers to the persistent correlation over time in the sequence of buy and sell orders within LOBs. Empirical studies have found that order signs exhibit significant autocorrelation, meaning that the direction of an order (whether it is a buy or a sell) is not random but influenced by the preceding sequence of orders. This phenomenon is often linked to the presence of strategic behavior by market participants, such as algorithmic traders, who base their decisions on the recent history of trades, thereby creating patterns that are detectable over extended periods. These patterns suggest that order flows are not entirely independent but are instead driven by underlying structures, such as the strategic placement of orders, herding behavior, or liquidity provision strategies.The identification of long-range autocorrelation in order signs has important implications for understanding market dynamics, particularly in relation to price formation. The presence of such autocorrelation indicates that information contained in the order flow is persistent – that is, information from orders long past may yet influence future trading behavior.
The persistent sign-autocorrelation in order flow can be explained primarily by two phenomena: herding and order-splitting. Herding occurs when traders, often influenced by market sentiment or the observed actions of others, tend to follow the prevailing trend. This behavior leads to a series of buy or sell orders in the same direction, contributing to the autocorrelation in order flow. In particular, herding is driven by the collective behavior of traders who interpret market signals similarly, resulting in a self-reinforcing cycle of orders that sustain a directional trend for a period.
Order-splitting, on the other hand, refers to large traders’ practice of breaking up substantial orders into smaller, sequential trades to minimize market impact and reduce transaction costs. This strategy naturally leads to autocorrelation in order flow, as a large buy or sell order is executed over time in smaller increments, creating a series of trades all in the same direction. Order-splitting is a deliberate tactic used by institutional traders and others managing significant positions to avoid revealing their intentions to the market and to achieve more favorable prices.
Empirical evidence suggests that while both herding and order-splitting contribute to the observed sign-autocorrelation in order flow, it is order-splitting that plays a more dominant role. Studies have shown that the autocorrelation patterns in order flow are more consistent with the gradual and systematic execution of large orders into smaller, less detectable orders (in other words, metaorders) rather than with the more sporadic and sentiment-driven herding behavior.46 Tóth, Palit, Lillo, and Farmer, Why Is Equity Order Flow so Persistent?. Journal of Economic Dynamics and Control (June, 2014). The persistence and regularity of autocorrelation align closely with the expected outcomes of order-splitting strategies, indicating that this is the primary driver behind the phenomenon.
Volatility clustering
Empirical research in market microstructure has consistently found that activity and volatility in financial markets tend to cluster, meaning that periods of high volatility are often followed by more high volatility, and periods of low volatility are followed by more low volatility. This phenomenon, known as volatility clustering, indicates that large price changes, whether positive or negative, tend to be followed by further large changes, while small price changes tend to be followed by small changes. This pattern suggests that volatility is not randomly distributed over time but rather exhibits persistence, where shocks to the market tend to have lasting effects on the level of volatility. This persistence has variously been attributed to the gradual dissemination of information, investor behavior, and market participants’ reactions to ongoing market conditions, which can perpetuate volatility for extended periods.
The clustering of volatility has significant implications for financial modeling, risk management, and trading strategies. Traditional models that assume constant volatility or independent price changes fail to capture this aspect of market behavior, leading to potential underestimation of risk during turbulent periods. In general, models where volatility in trading activity is assumed to be either constant, or to evolve relatively smoothly according to some ad hoc stochastic function, will underestimate the degree of volatility or activity that could potentially emerge all of a sudden, seemingly without warning.
The theoretical victims of this empirical result are quite numerous. Perhaps the highest-profile one would be the Black-Scholes Options Pricing model, which counts among its many assumptions constant volatility of the underlying asset over time. As discussed in a previous footnote, this assumption simplifies the mathematics of option pricing but fails to capture the reality of financial markets where volatility often clusters. Brownian motion (recall its importance to the Bachelier model, as well as Black-Scholes, which in many ways was the modern inheritor of Bachelier’s insights) is a stochastic process that has been used to model asset prices in finance since the beginning of the twentieth century. Crucially, it assumes that the returns of an asset follow a normal distribution with constant drift and volatility. While Brownian motion is mathematically convenient and forms the basis for models like Black-Scholes, though, it does not account for volatility clustering observed in real markets, where periods of calm are often followed by turbulence, leading to fat tails and other deviations from normality in asset return distributions.
As a result, models like GARCH (Generalized Autoregressive Conditional Heteroskedasticity) have been developed to better capture this clustering by allowing volatility to evolve over time based on past market conditions.47 For a general discussion of GARCH models and their use in econometrics (with particular focus on applications for financial time series data), see Engle RF. “GARCH 101: The Use of ARCH/GARCH Models in Applied Econometrics”. J Econ Perspect. 2001;15(4):157-168. doi:10.1257/jep.15.4.157. A very common variant is the Garch(1,1) model. The “(1,1)” refers to the number of lagged terms used in the model. The first “1” represents the number of lagged squared returns (ARCH term), and the second “1” represents the number of lagged variances (a GARCH term) used to predict the current period’s variance. A GARCH(1,1) model is a time series model that captures volatility clustering by modeling the current period’s variance as a function of the previous period’s variance and the squared return shock. It combines the effects of past squared returns (ARCH term) and past variances (GARCH term), making it useful for predicting changing levels of volatility over time in financial markets.
Order flow theorists have attempted to take volatility clustering (as well as others of the main statistical regularities we’re presently discussing) into account quite cleverly by deploying a class of equations called Hawkes processes.
Hawkes processes were originally inspired by the study of earthquakes, where the occurrence of one earthquake can trigger subsequent aftershocks, leading to a clustering of seismic events over time. In essence, a Hawkes process is a self-exciting point process,48 A self-exciting point process is a way to model events that happen over time, where each event increases the chances that more events will happen soon after. Imagine you’re at a party, and someone starts clapping. This initial clap might encourage others to start clapping too, which then leads to even more people joining in. The clapping spreads because each clap makes it more likely that the next one will happen – eventually, once enough people have clapped for long enough, the clapping begins to die down, returning to a clap-less lull (potentially setting up future clapping events, though there’s no sure-fire way to known when). In a self-exciting point process, the occurrence of one event (like a clap) makes other similar events (more claps) more likely to occur in the near future. meaning that each event increases the likelihood of future events occurring in the near term. This concept was first developed by Alan Hawkes in the early 1970s to model the temporal distribution of earthquakes, where aftershocks tend to follow a main shock, creating clusters of seismic activity.
In financial markets, Hawkes processes have been applied to study volatility clustering. This clustering can be seen as analogous to the way aftershocks follow an initial earthquake. In this context, each “event” in the Hawkes process might represent a significant market move or trade, which increases the probability of further market activity in the near future. The self-exciting nature49 For an in depth discussion of self-exciting behavior in financial markets in general, seek out Hainaut D, Moraux F. “A multifactor self-exciting jump diffusion approach for modeling the clustering of jumps in equity returns.” (2016). of the Hawkes process helps to capture the way that volatility tends to cluster, with bursts of high volatility followed by periods of relative calm.
However, linear Hawkes models, while useful, have limitations when applied to financial data. One key limitation is that they don’t fully capture the strong volatility clustering observed in real financial markets. Additionally, these models, at least in their basic, linear form, struggle to reproduce the fat-tailed distribution of returns50 Fat-tailed distributions refer to probability distributions that exhibit heavier tails than the normal distribution, meaning that extreme events (large positive or negative returns) are more likely to occur than would be predicted by a normal distribution. This phenomenon has been empirically confirmed in financial markets, where asset returns frequently deviate from the normal distribution, exhibiting higher kurtosis and skewness. Notable empirical studies include Fama’s (1965) seminal work on stock market prices, which demonstrated that daily returns exhibit significant kurtosis and deviate from the normal distribution, and Mandelbrot’s (1963) analysis, which introduced the concept of stable Paretian distributions to describe asset returns. More recent studies, such as Cont (2001), have further confirmed the prevalence of fat-tailed distributions across various financial markets, highlighting the need for models that account for these extreme events when analyzing risk and returns. – where extreme price movements are more common than would be expected under a normal distribution. These limitations arise because the linear Hawkes process assumes a simple, direct feedback mechanism.
To address these shortcomings, researchers have extended Hawkes processes by incorporating additional effects, such as the impact of price returns51 A price return is the percentage change in the value of an asset over a specific period, reflecting the difference between its current market price and a previous market price (without accounting for dividends or interest payments). in the feedback mechanism. By doing so, the models can better replicate the empirical properties of financial markets, such as more pronounced volatility clustering and the fat-tailed distribution of returns. These enhancements allow Hawkes models to more accurately reflect the intricacies of market behavior, making them a more powerful tool for analyzing financial data.
Almgren-Chriss Optimal Execution Model (2000)
The Almgren-Chriss model, introduced by Robert Almgren and Neil Chriss, is a foundational framework for the optimal execution of large financial orders. It addresses the need to minimize the costs associated with executing large trades by considering both the temporary and permanent impacts of trading on market prices. The model is particularly notable for its use of a square-root market impact function, which contrasts with earlier models that often assumed a linear relationship between trade size and price impact. By incorporating a square-root function, the Almgren-Chriss model provides a more realistic depiction of how large trades influence market prices, especially in highly liquid markets.
The square-root impact assumption in the Almgren-Chriss model is a critical innovation. Empirical evidence from financial markets suggests that the price impact of trades does not increase linearly with trade size; instead, it increases at a diminishing rate. The square-root function reflects this reality, capturing the nonlinear nature of market impact: as the size of the trade increases, the incremental cost of additional trading decreases. This operationalization contrasts with linear models, where doubling the size of a trade would double the impact, potentially overestimating costs for large orders and underestimating them for small ones. The square-root function better aligns with observed trading behavior, where large orders tend to be broken down into smaller ones to mitigate impact.
In practical terms, the Almgren-Chriss model enables traders to determine the optimal execution strategy by balancing the trade-off between minimizing market impact costs and minimizing exposure to price volatility. The model’s equations provide a path for executing large orders over time, suggesting how to split the total order into smaller pieces to reduce market impact while still achieving the trade within a reasonable timeframe. This approach has become a standard in the field, influencing both theoretical research and practical applications in algorithmic trading and portfolio management.
The Almgren-Chriss model has served as a pivotal reference point for market microstructure researchers since its introduction, profoundly influencing the field’s understanding of market impact and optimal trade execution. By operationalizing the square-root impact law, it provided a more accurate and empirically validated approach to modeling how large orders affect market prices, shifting the paradigm from simpler linear models. The model’s framework for balancing market impact costs with price volatility risk has been extensively cited and expanded upon in subsequent research, driving innovations in algorithmic trading, execution strategies, and the broader study of market dynamics. Its influence is evident in the way modern trading algorithms are designed to optimize trade execution by minimizing cost.
Traders had long intuited that the relationship between order size and market price was something of the nature of a square-root function – and that metaorders were a viable strategy. But the Almgren-Chriss model gave that anecdotal experience the backing of hard empirical data as well as theoretical coherence. In short, it created a scientific basis for the analysis, and even the design, of large-scale, effective metaorders.
The Shape of Market Impact
Unfortunately – or maybe fortunately – the Almgren-Chriss model had bigger consequences for mainstream theory as well.
Remember that Black-Scholes model we mentioned before as an heir to Bachelier’s model? Let’s go into it in more detail. The Black-Scholes option52 Briefly, an option is a type of security that is a contract granting the buyer a right, but in no way an obligation, to undertake an action involving the buying or selling of an asset at a particular price before a certain date. In the two major types of options, one is a call option, which enables the buying of the asset, and another one is a put option, which enables its selling. Where the strike price of an option is one predetermined by its writer/seller at which the buyer can buy (in the case of a call) or sell (in the case of a put) the underlying asset, the premium or market price of the option refers to the price one pays to purchase the option, which would fluctuate based on the dynamics of the order flow through the LOB. The two main kinds of options are European and American. The key distinction between European-style and American-style options lies in the fact that, while European options can be exercised only at expiration, American options can be exercised at any time before or at expiration. pricing model (in its original form) calculates the theoretical price of a European-style option by considering five key variables: the current price of the underlying asset, the option’s strike price, the time to expiration, the risk-free interest rate, and the asset’s volatility. The model assumes that the price of the underlying asset follows a geometric Brownian motion, meaning it fluctuates in a continuous, random way with a constant volatility. Using these inputs, the Black-Scholes equation generates a “risk-neutral” valuation, implying that investors are indifferent to risk, allowing the future price of the asset to be discounted at the risk-free rate. The central part of the model relies on a partial differential equation (PDE),53 In general, PDEs describe how a function changes in more than one variable simultaneously. They are termed “partial” because they involve partial derivatives, which signal the rate of change of a function with respect to one variable while keeping the others fixed. These equations model systems in which more than one factor acts on an outcome, whether it be the conduction of heat through a medium, the movement of waves, or fluid dynamics. A PDE involves a relation between partial derivatives of a function and hence is useful for the description of phenomena in which change is produced over time or space, or most generally in several variables independently. which represents the evolution of the option’s price over time and can be solved to obtain a closed-form formula for both call and put options. In practical terms, the model also assumes that there are no transaction costs, no dividends on the underlying stock paid during the option’s life, and that the markets are perfectly efficient (in a market-efficiency sense). Although the model simplifies real-world conditions, it provides an important benchmark (important in the sense that it’s widely used) for pricing options by balancing the factors of time decay, volatility, and intrinsic value.54 If this seems tautological, then you’re not alone. It is certainly odd to think about how, regardless of whether its assumptions are true, a model that many people use for pricing options ends up becoming relevant to options prices just because a lot of people use it to price options. That is, a bunch of people using the same model produce roughly the same result, regardless of the quality of the model. Keep this somewhat counterintuitive idea in your head, as it’s going to become important later in the essay! A key insight from the model is the concept of hedging, where a portfolio of the underlying asset and the option can be maintained in a risk-free state by continuously adjusting its composition, known as delta hedging. In simpler terms, the Black-Scholes model offers a mathematical way to estimate the fair value of an option, though as we can see, it simplifies some market realities into terms more digestible by well-behaved PDEs.
The square root impact result55 Often referred to as a “law” to denote its empirical strength, the square root law in finance refers to the empirical observation that the market impact of trading a large order increases with the square root of the order size. Specifically, if I(Q) represents the market impact, where Q is the order size, the relationship is given by I(Q) ∝ σ √Q, where σ is the volatility of the asset. This implies that doubling the size of an order will not double the market impact, but rather increase it by a factor of √2. The law captures the idea that larger trades tend to have a disproportionately smaller impact on the price than smaller trades on average, reflecting market liquidity and participants’ ability to absorb large trades without excessive price shifts – in other words, this relationship is a reflection of an inherent resilience within order flow across a wide range of outcomes with the LOB’s dynamics. The square root law is often used in optimal execution strategies (like Almgren-Chriss) to minimize trading costs and reduce market impact. When institutional traders use metaorders to execute large buys or sells, they are utilizing this empirical result to plan out how to execute their large order while controlling for the cost of execution. For a more in depth discussion of the law and its domains of validity, seek out Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, Quotes and Prices. 2018. Cambridge University Press, pp. 233-243. presented a serious problem for the Black-Scholes option pricing model because the model relies on the assumption that markets are frictionless and that the execution of trades, particularly the continuous rebalancing required for delta hedging,56 When you own an option (which gives you the right to buy or sell an asset at a specific price), the value of that option changes as the price of the underlying asset moves. Delta is a measure of how much the price of the option is expected to change if the price of the underlying asset changes by a small amount. If you have a delta of 0.5, for example, it means that for every $1 change in the price of the underlying asset, the price of the option would change by $0.50. Delta hedging involves balancing your position in the option with an opposite position in the underlying asset to reduce risk. If you own a call option (which benefits from the asset price going up) and you want to hedge against the possibility that the price might go down, you could sell some of the underlying asset. The amount you sell would be based on the delta. If the delta is 0.5, you might sell half as much of the asset as the option controls. This way, if the asset price moves, the losses on one side (the option) are offset by gains on the other side (the asset), keeping your overall position more stable. In essence, delta hedging helps you manage the risk of price movements by creating a balanced position that minimizes the impact of those movements on your portfolio. You can think of it like purchasing a temporary insurance policy against your underlying position. can occur without influencing the underlying asset’s price. The discovery that price impact follows a square root law implied that large trades, which would be necessary for delta hedging by large institutional traders in practice, could significantly move prices in a nonlinear fashion. This nonlinear impact on prices contradicted the Black-Scholes assumption of continuous, costless trading, leading to potential mispricing of options (relative to some trend in price they’ve spotted, whether real or imagined) and a need to account for liquidity and market impact costs in the model. As a result, the square root impact finding highlighted a critical limitation in the Black-Scholes framework, especially in real-world trading, where large orders are executed and markets are not perfectly liquid.
Study after study has reproduced this result, to the point that it has become known as one of the most verifiable facts in all of quantitative finance. To illustrate this, here is a table of 10 recent papers confirming a square root impact relationship with trading size across various asset classes (including bitcoin) in recent years:
| Title | Author(s) | Journal | Year |
|---|---|---|---|
| Market Impact: Empirical Evidence, Theory and Practice | Said E, Ayed ABH, Thillou D, Rabeyrin JJ, Abergel F | Quantitative Finance | 2022 |
| Crossover from Linear to Square-Root Market Impact | Bucci F, Benzaquen M, Lillo F, Bouchaud JP | Physical Review Letters | 2022 |
| The Square-root impact law also holds for option markets | Tóth B, Eisler Z, Bouchaud JP. | Quantitative Finance | 2016 |
| How efficiency shapes market impact | Farmer JD, Gerig A, Lillo F, Waelbroeck H. | Quantitative Finance | 2013 |
| Empirical Market Microstructure: The Institutions, Economics, and Econometrics of Securities Trading | Joel Hasbrouck | Oxford University Press | 2007 |
| A Million Metaorder Analysis of Market Impact on the Bitcoin | Donier & Bonart | Market Microstructure and Liquidity | 2014 |
| Anomalous price impact and the critical nature of liquidity in financial markets | Tóth et al. | American Physical Society | 2011 |
| A theory of power-law distributions in financial market fluctuations | Gabaix et al. | Nature | 2003 |
| Measuring and Modelling Execution Cost and Risk | Engle, R., Fernstenberg, R., & Russell, J. | The Journal of Portfolio Management | 2012 |
| Large Bets and Stock Market Crashes | Kyle, A.S., & Obizhaeva, A. A. | Review of Finance | 2016 |
What could be causing this square-root shape of impact? There are several leading theories. One suggests that the square-root law of market impact derives from the fact that the price impact of trading increases with the size of the order, but at a diminishing rate. This phenomenon can be traced back to the local linearity of the order book within very small price ranges. When a trade is executed, it first impacts prices linearly within a very narrow range where liquidity is relatively dense (for example, right at and near the best bids and asks). However, as the trade size grows, it spans across multiple price levels where liquidity may be less dense, resulting in a sublinear aggregation of local impacts, which could give rise to the overall, aggregate square-root relationship.
So, in other words, the key transition from local linearity to a square-root impact occurs because larger trades must consume liquidity at increasingly distant price levels, where the order book depth typically thins out.57 Moro, E., Vicente, J., Moyano, L. G., Gerig, A., Farmer, J. D., Vaglica, G., Lillo, F., & Mantegna, R. N. (2009). Market Impact and Trading Profile of Large Trading Orders in Stock Markets. Physical Review E, 80(6), 066102. This thinning means that the additional price impact per unit of traded volume diminishes as the trade size increases. Essentially, while each small portion of the trade might have a linear effect, the cumulative impact becomes sublinear due to the varying density of the order book, resulting in the observed square-root shape.
In a similar vein, Jean-Philippe Bouchaud’s “liquidity memory time” explanation for the square-root impact law revolves around the idea that market liquidity has a memory or persistence over time.58 Bouchaud, J.-P., Farmer, J. D., & Lillo, F. (2009). How Markets Slowly Digest Changes in Supply and Demand. Handbook of Financial Markets: Dynamics and Evolution, pp. 57-160. Elsevier. When a trade occurs, it temporarily depletes liquidity at certain price levels, but this liquidity gradually recovers as other traders place new orders with knowledge of the past depletion. In this formulation, the square-root impact law emerges because the market’s response to large trades depends on how quickly liquidity replenishes, which is influenced by this memory effect. If liquidity recovers slowly, the cumulative impact of sequential trades becomes less than linear. The idea is that the memory of past trades and the gradual recovery of liquidity create a nonlinear effect, explaining the square-root relationship between order size and price impact.
Recently, order flow theorists at Kyoto University added to the already-vast empirical literature with their own study confirming the sublinear relationship, but decided to go a step further and claim (based on their analysis of an expanded dataset from the Tokyo stock exchange) that this relationship holds across a wide enough range of securities and time scales that we might as well consider it to be a universal scaling feature of financial markets.59 Sato, Y., & Kanazawa, K. (pre-print. November 21, 2024). Does the square-root price impact law belong to the strict universal scalings?: quantitative support by a complete survey of the Tokyo stock exchange market. arXiv.org. https://arxiv.org/abs/2411.13965 Personally, I have some trouble accepting any “laws” or “universals” when it comes to the social sciences, but it is nonetheless interesting to see how better access to data over time allows the theorists to make such claims with sufficient rigor to be tested.
Self-Referentiality and the Inherent Endogeneity of Financial Markets
Empirical studies in financial markets have increasingly confirmed that the majority of price variance is driven by endogenous factors rather than external news or macroeconomic shocks. These endogenous factors include the interactions among traders, the dynamics of order flows, and the microstructure of the market itself, such as the placement and cancellation of orders within the limit order book. Research has shown that price movements often result from the feedback loops created by the trading activities of market participants, where actions like the execution of large metaorders can trigger cascades of subsequent trades, amplifying price changes. As much as 80% of price changes are induced by self-referential (or endogenous) effects.60 Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, Quotes and Prices. 2018. Cambridge University Press, p. 36 This self-reinforcing behavior within the market’s internal mechanisms suggests that much of its observed price volatility is generated by its own internal processes rather than exogenous information. Consequently, market researchers’ focus has shifted toward modeling and predicting these endogenous dynamics to better understand and manage price fluctuations.
Returning to Hawkes processes for a moment, one interesting finding after calibrating the algorithm to financial data is that the branching ratio61 In the context of a Hawkes process, the branching ratio is a parameter that quantifies the degree of self-excitation or clustering in the process. Mathematically, it represents the expected number of offspring (or triggered events) produced by a single event. If the branching ratio is less than 1, the process is subcritical, meaning that events tend to die out over time. If it equals 1, the process is critical, leading to a balance between events triggering and ceasing. If the branching ratio is greater than 1, the process is supercritical, indicating that events tend to proliferate, potentially leading to an explosive growth of events over time. This ratio is fundamental to determining the long-term behavior and stability of a given Hawkes process. comes in very close to 1, implying that each event almost, but not quite, triggers one subsequent event on average. This suggests a near-critical state to the system. In other words, financial markets are highly self-exciting:events (trades) are likely to lead to a sustained chain of further events, but the system is just stable enough that it doesn’t explode into an infinite number of events.
In practical terms, a branching ratio close to 1 indicates that the system is in a delicate balance. Events tend to cluster together, and these clusters can last for a long time, but eventually they die out. This could represent a situation where a single large trade or price move leads to a series of reactions and further trades, creating the volatility clusters we see in actual practice.
Cont-Bouchaud (2000)
In the preceding sections, we’ve briefly explored Jean-Philippe Bouchaud’s analysis of the predominance of endogenous effects on price changes in financial markets, one of the major empirical discoveries that resulted from the advent of limit order book databases. If it was true that most of the factors affecting financial markets were endogenous, though, then anyone hoping to predict market behavior would need to be able to identify these various factors and understand how they interacted with each other. Bouchaud and his colleague Rama Cont would be among the first to try to answer this question. The resulting Cont-Bouchaud model62 Cont, R., & Bouchaud, J.-P.: “Herd Behavior and Aggregate Fluctuations in Financial Markets.” Macroeconomic Dynamics (2000). is a prominent agent-based framework developed to analyze the dynamics of limit order books in financial markets.
An important distinctive feature of this model is its capability to emulate complex behaviors by a diverse set of market participants, also called “heterogeneous agents.” Agents in the model include both liquidity providers who place limit orders and liquidity takers who execute market orders. Its main objective is to model the microstructure of financial markets by studying how interactions among these agents shape the order book and, therefore, influence key market variables such as price formation, volatility, and liquidity. The Cont-Bouchaud model simulates the decision-making process of the agents in detail and hence gives a bottoms-up view of how market dynamics emerge from the joint actions of individual traders.
Central to the Cont-Bouchaud model are a number of key assumptions identifying the behavior of agents and the market structure. One is that agents are independent, each acting in their own interest according to personal choice, different information available, and risk appetite. This supposes that liquidity providers set limit orders in order to capture the bid-ask spread or to hedge present positions, whereas liquidity takers will strive to execute their trades immediately. These agents are modeled under the assumption of bounded rationality; that is, they do not have perfect foresight and do not have complete information; instead, they operate under the use of heuristics or simple decision rules. The order book itself is modeled as a central repository where all limit orders are stored until they are either executed or canceled. At any instant, the best bid and ask represent the most competitive prices available in the market.
Another significant assumption of the Cont-Bouchaud model is the random nature of order arrivals and cancellations, reflecting the stochastic environment of real-world trading. The model typically assumes that order arrival rates and cancellation rates, as well as the size of orders, follow specific statistical distributions, often derived from empirical market data. This randomness introduces variability into the system, leading to fluctuations in liquidity, price changes, and volatility. Importantly, the Cont-Bouchaud model does not assume market efficiency; instead, it allows for the emergence of phenomena like volatility clustering and fat-tailed return distributions, which are commonly observed in actual markets but are difficult to capture with traditional models. By incorporating these assumptions, the Cont-Bouchaud model provides a flexible and somewhat realistic framework for understanding the complex and often nonlinear behaviors of the LOB.
The Cont-Bouchaud model has generally held up okay in light of the empirical findings of the past 20 years. On the positive side, the model’s ability to capture the emergent properties of financial markets, such as the formation of bid-ask spreads, volatility clustering, and the fat-tailed distribution of returns, aligns well with empirical observations. The assumption of heterogeneous agents with different strategies and the stochastic nature of order arrivals and cancellations has proven to be a robust framework for explaining many aspects of market behavior, particularly in highly liquid markets.
However, the model has also faced challenges, and in several places it has been extended or refined to better match empirical data. For instance, the original Cont-Bouchaud model tends to oversimplify some aspects of trader behavior and market structure, such as the role of institutional traders, the impact of high-frequency trading, and the complexities of order routing and execution in modern electronic markets. Empirical studies have shown that factors like order flow imbalance, liquidity fluctuations, and the influence of large metaorders require more nuanced modeling approaches that go beyond the assumptions of the original framework. As a result, extensions of the Cont-Bouchaud model have been developed to incorporate these factors, generating a more accurate representation of market dynamics.
Overall, while the Cont-Bouchaud model (and its close variants) remains a foundational tool in market microstructure research, in no small part because it has proven capable of evolving alongside new empirical findings. Researchers have built on its core principles, integrating more sophisticated agent behaviors and market mechanisms to address the complexities observed in real markets.
Liquidity, Revisited:
Latent Liquidity and our Collective Hidden Intentions
Recall from our look into liquidity in financial markets that the fundamental unit of analysis there was the availability of priced orders in LOBs. This definition works just fine for orders already placed (and visible) within the LOB, but what of orders not yet placed? Surely our intentions for orders matter for the evolution of the LOB as well? This intuition, one long held by traders and researchers alike, has been formalized in a relatively new concept called latent liquidity.
Latent liquidity in market microstructure refers to the hidden or unobservable liquidity that exists in a market but is not immediately visible in the limit order book. It represents the potential willingness of market participants to trade at certain prices, but the corresponding orders are not displayed or committed until market conditions trigger their execution. Latent liquidity can come from large institutional investors or algorithmic traders who wait for specific signals or conditions before placing their orders to avoid revealing their intentions to the market. This hidden liquidity can significantly impact price dynamics and market stability, as it may suddenly materialize in response to price movements or other market events, thereby influencing the availability of liquidity and the execution of other trades. Understanding latent liquidity is crucial for market participants, as it affects their ability to gauge true market depth.
There are many different models of latent liquidity, but one method common to many of them is to essentially construct a much larger LOB that captures, at least theoretically, both the intentions already visible in the LOB in the form of placed orders in the queue, as well as the (perhaps fundamentally?) unobservable orders that exist mostly in the minds of participants waiting out developments in the LOB.
There are many valid reasons to posit the existence of something like latent liquidity. For instance, it is not uncommon for institutional traders and other large participants to attempt to trade a relatively large percentage of the total market capitalization of a company within a single metaorder, as we’ve seen.63 Market capitalization is calculated by multiplying the total number of shares of a security outstanding (the number of shares issued and sold, excluding those held by the company itself) by the current market price of the stock. In a limit order book, the notional market cap of the priced orders (the limit order prices multiplied by the volumes at each price level) often represents only a small percentage of the company’s overall market capitalization. Because the total outstanding volume shown in the LOB at any given time is typically very small relative to the total market cap of the company (often less than 0.1%), and because the average daily traded volume is typically also quite small relative to proposed size of these institutional orders, making such a large trade can quickly become uncomfortable for the institutional trader, who doubtless has calculated the expected price impact their metaorder would have if they executed it over the course of just one trading day, and practically gasped. Therefore, many order flow theorists have assumed, reasonably perhaps, that at any given time a good deal of both the demand and supply for a given security remains unobservable.64 Jean-Philippe Bouchaud, Julius Bonart, Jonathan Donier, and Martin Gould. Trades, Quotes and Prices. 2018. Cambridge University Press, pp. 337-338
Diffusion-Reaction Models
Modeling the latent order book (and its interactions with the visible LOB) has become a complex issue among order flow theorists. One common approach is to construct two separate LOBs: one for the visible orders, which we’ve explored so far, and another for the unobserved or latent orders. The next step involves modeling a transition function that describes how unobserved orders diffuse into the visible LOB. These unobserved orders are often treated as a random, or stochastic, function related to changes in the state of the visible LOB. A recent example of such a model is the diffusion-reaction model. Inspired by models of chemical processes, these financial models typically include three main components: the arrival of new orders (often generated stochastically), the resulting revision (or repricing) of existing orders in the visible LOB, and the cancellation of existing orders in response to either the new orders or the repricing from revisions. The order revision component – often the focal point of these models – is typically a function of discrete changes in market price as the LOB transitions from one state to the next, with sensitivity coefficients usually generated stochastically according to carefully selected parameters.
Reaction-diffusion models offer a flexible framework with many useful properties, including the ability to account for heterogeneous beliefs among agents. This approach seemingly captures one of the statistical regularities in LOBs that we’ve discussed: the dynamic feedback loop that characterizes order flow changes in an LOB. However, diffusion-reaction models have a significant drawback. Like the chemical reactions they’re modeled after, without significant modifications, these models tend to resolve prices towards an equilibrium or steady state. While this makes sense in chemistry, where the “agents” – the molecules – have homogeneous revision functions, and their interactions lead to more stable energy states, it’s less applicable to financial markets. Given the high degree of intermittency, clustering of volatility and activity, and the unpredictability of these spikes (not to mention the fat-tailed distribution of returns), it’s difficult to argue that price behaves as an equilibrating process. In fact, one could go even further in saying that the existence of a large pool of un-submitted orders in the form of a latent order book (essentially what the latent liquidity concept is) by itself calls into question the validity of equilibrium models generally. The “assume steady state” approach may work in econometrics or engineering courses, a handy concept for students to confidently invoke during an exam on thermodynamics or what have you, but in my opinion, it’s inadequate for advancing our understanding of the dynamic feedback loops and inherent nonlinearities of financial markets.
Machine Learning Models of LOBs
In recent years, machine learning researchers have focused some of their deep learning models on trade-level data from LOBs, in order to identify and track the appearance (or reappearance) of key features – most notably, price data for the purposes of price trend prediction.
Back-testing is a method used to evaluate trading strategies by simulating them on historical data, particularly on models of the LOB. This approach allows traders to analyze how their strategies would have performed in past market conditions. However, a significant limitation of back-testing is that it does not account for the market’s reaction to the trades being simulated, leading to potentially misleading results. As an alternative, a Generative Adversarial Networks (GANs) approach from machine learning offers a more dynamic solution.65 Nagy, Peer & Frey, Sascha & Sapora, Silvia & Li, Kang & Calinescu, Anisoara & Zohren, Stefan & Foerster, Jakob. (2023). Generative AI for End-to-End Limit Order Book Modelling: A Token-Level Autoregressive Generative Model of Message Flow Using a Deep State Space Network. 10.48550/arXiv.2309.00638. By generating realistic market responses to trades in a simulated environment, GANs can better capture the interactions between a trading strategy and the market, providing a more accurate assessment of the strategy’s effectiveness.
A well-known GAN model specifically designed for simulating LOBs is the appropriately-named LOBGAN model. This model employs a Conditional GAN (CGAN) approach to generate realistic LOB data by conditioning itself on the current state of the market.66 Coletta, Andrea, Joseph Jerome, Rahul Savani and Svitlana Vyetrenko. “Conditional Generators for Limit Order Book Environments: Explainability, Challenges, and Robustness.” Proceedings of the Fourth ACM International Conference on AI in Finance (2023): n. pag. The CGAN framework allows the model to generate new hypothetical orders that realistically reflect how the market would respond to certain trading activities, thereby overcoming the limitations of traditional back-testing, which does not account for market reactions to simulated trades.
LOBGAN integrates with market simulators to create an interactive environment where trading strategies can be tested against dynamically generated market conditions, making it a powerful tool for both research and practical applications in financial markets. This model has been shown to effectively replicate key statistical properties of real markets, known as “stylized facts,” such as volatility clustering and heavy-tailed distributions of returns, which are critical for ensuring the realism of simulated trading environments.
A deep learning model is a type of artificial neural network composed of multiple layers of interconnected nodes, or “neurons,” inspired by the human brain’s structure. These models excel at capturing complex patterns and relationships in large datasets, making them useful for tasks like image recognition, natural language processing, and financial modeling. Some deep learning models have been shown to be capable of modeling the intricate and dynamic changes in LOB states by processing historical trade data (using trade-level databases like LOBSTER) and extracting relevant features for predicting future price movements.67 Prata, M., Masi, G., Berti, L. et al. “Lob-based deep learning models for stock price trend prediction: a benchmark study”. Artif Intell Rev 57, 116 (2024). As we’ve seen, LOB price data is both extremely noisy and nonstationary – a perfect candidate for analysis by deep learning models.68 D. T. Tran, N. Passalis, A. Tefas, M. Gabbouj and A. Iosifidis, “Attention-Based Neural Bag-of-Features Learning for Sequence Data,” in IEEE Access, vol. 10, pp. 45542-45552, 2022
Putting it All Together
As we’ve observed, standard Gaussian random-walk models – such as those proposed by Bachelier, Black-Scholes, and the efficient-market hypothesis (EMH) – significantly underestimate extreme price fluctuations. In reality, price changes follow fat-tailed distributions, where extreme events are more common than Gaussian models predict. Market activity and volatility are highly intermittent and clustered, with periods of intense activity often occurring independently of exogenous information, such as the arrival of news regarding company performance, that standard economic theory deems fundamental. Most market activity is endogenous, triggered by past trading activity. A nearly flat volatility signature plot, combined with evidence of strong self-referentiality and endogeneity, challenges the validity of the EMH. Signature plots over timescales from seconds to months are remarkably flat, indicating that while markets may not be fundamentally efficient, they are statistically efficient,69 A signature plot is a graphical representation used in time series analysis to examine how certain statistical properties – such as volatility, variance, or autocorrelation – change with different timescales. Typically, this plot displays a measure like the standard deviation of returns against the length of the time interval on a log-log scale. In many stochastic processes, especially those resembling Brownian motion or random walks, these measures exhibit specific scaling behaviors. For example, in an ideal random walk, the standard deviation of returns scales with the square root of time, resulting in a straight line with a slope of 0.5 (on the log-log plot). A flat signature plot occurs when the plotted line is horizontal, indicating that the statistical measure remains constant across different time scales. This flatness implies that the expected scaling behavior is absent. In practical terms, a flat signature plot suggests that the time series lacks temporal dependence or multi-scale structure. In the context of financial markets, such a plot might reveal that price changes are dominated by high-frequency noise or microstructural effects. exhibiting excess volatility beyond what traditional models, like Black-Scholes or the EMH, suggest. If one views the long run as merely an accumulation of short-run events, these findings are not just troubling – they are devastating for the economic concept of value in asset pricing. The long-run volatility of the market appears to be determined largely by higher-frequency behavior, as shown in volatility signature plot studies.
An alternative perspective, motivated by a microstructural viewpoint, suggests that highly optimized execution algorithms, used by strategically behaving participants, actively search for detectable correlations or trends in pricing behavior, whether real or spurious. In the standard economic view – held by both mainstream and at least some heterodox economists – markets are expected to reflect the true value of assets, with prices being merely unpredictable and diffusive deviations from these true values. However, countless empirical studies have now challenged this view with several fundamental puzzles, including excess trading, excess volatility, and the persistence of trend-following behavior. Mainstream economists have tried to address the excess trading puzzle by introducing noise traders into their models, but the other puzzles remain unresolved. A competing vision – the one expanded on in this essay – proposes that prices are driven by order flow, regardless of its informational content. In this view, markets are more about predicting the behavior of other traders than reflecting anything fundamental about what’s being traded. This order-driven perspective may explain why markets are so endogenous and volatile, leading to what proponents of a concept of fundamental value would consider “mispricings.” Due to price impact and the interactions of strategic behavior by heterogeneous agents, the predictability these agents seek is often ironed out, resulting in prices that, while not efficient in the EMH sense, are highly statistically efficient.
Part IV:
A Modest Contribution:
One Notion, Two Models
One of the key problems of developing self-referential, endogenous order-driven models of price formation in LOBs is the sheer complexity of the problem.70 Rosu, Ioanid, “A Dynamic Model of the Limit Order Book” (June 6, 2008). Review of Financial Studies, Vol. 22, pp. 4601-4641, 2009.
In developing my models below for price formation in financial markets, I am going to begin from first principles, using the method of grounded theory,71 Grounded theory is a qualitative research method that involves the systematic collection and analysis of data to generate theories that are grounded in the observed data itself. Unlike other research approaches that begin with a hypothesis or a pre-existing theory, grounded theory develops theoretical frameworks directly from the data through an iterative process. Researchers start by collecting rich, detailed data through methods such as interviews, observations, or document analysis. As data is gathered, it is coded to identify patterns, concepts, and categories, which are then refined and connected through constant comparison – continuously comparing new data with existing codes and categories. This process involves moving back and forth between data collection and analysis, with emerging insights guiding further data collection. The goal is to develop a theory that is closely linked to the data, offering an explanatory framework that is directly relevant to the context being studied. Grounded theory is particularly useful for exploring new or complex phenomena where existing theories may be inadequate. drawing as best I can from the empirical observations painstakingly assembled by market microstructure researchers and the order flow theorists who followed them.
The Argument
Limit order books are, first and foremost, data structures.72 Here, I mean a data structure specifically in the computer science sense of the term: it is a specialized format for organizing, managing, and storing data in a way that enables efficient access and modification. Data structures are fundamental building blocks used to implement algorithms and optimize the processing of large and complex datasets. Examples of data structures include arrays, linked lists, trees, graphs – and most importantly for our purposes here – queues. The flow of orders into and out of the LOBs constantly updates their states, constituting a data-generating process.73 In this instance, I am borrowing this term from statistics. A data-generating process in statistics refers to the underlying mechanism or model that produces some observed data. It describes how variables are related, the distribution of those variables, and any randomness or noise in the data, serving as a conceptual framework for understanding and predicting the patterns and outcomes in a dataset. Most of the elements that market participants care about when deciding whether to place a new order are present in the LOB and the evolution of its state through time – volume, liquidity, arrival time, executions, cancellations, and of course, price.
The order flow generated by this process represents the trading activity that ultimately drives price fluctuations, through the complex dynamics of the LOB we’ve discussed in detail. We can therefore say, with precision, that price formation is a consequence of order flow.
But what drives order flow? It’s one thing to examine how price changes occur at the microstructural level when limit orders are matched with market orders, and then to explore why this process is complex and nonlinear. However, we must also consider the more challenging question of the market participants themselves and the beliefs that motivate their trading strategies in the first place.
We’ve established that trading activity generates market prices with each execution, creating a price history that participants can reference. Naturally, market participants will analyze this history and form opinions about potential future prices. While each element of the limit order book – such as volume, submission time, and cancellation rate – is doubtless being analyzed and incorporated into many participants’ models, the primary concern remains: what prices prevail now and in the future? Any trader’s goal is to achieve a positive, risk-adjusted return – in other words, to make money.
As noted throughout this essay, both trading activity and price volatility are strongly clustered in time and are intermittent. These clustering events appear to be driven largely, if not entirely, by endogenous outcomes within the LOB. The arrival of new orders – and the subsequent reactions from participants, including price revisions or cancellations of existing orders – seems to be triggered by changes made to the LOB state, through self-referential or self-excitation processes.
There is considerable evidence that strategic behavior among participants (who can be modeled as agents with heterogeneous qualities) plays a significant role in the behavior of the LOB. This includes the submission of large metaorders over time by institutional traders and execution services firms on behalf of clients, as well as the reactions to – or anticipations of – these metaorders by other participants, such as high-frequency trading firms. The complex interactions between these groups can lead to important and long-lasting changes in LOB behavior. This is evident in the persistence of order direction – buys followed by more buys, or sells followed by more sells – juxtaposed with the relative unpredictability of price changes. The direction of orders is predictable, while the execution price is not.
A continuous game of cat-and-mouse seems to exist between those tasked with buying or selling large quantities of securities over time and those waiting to detect these large orders, hoping to take advantage by trading on the other side – thereby, at times, causing price to trend noticeably. This could lead to price improvement for the metaorder detector, such as the high-frequency trading firm that was lying in wait, and, conversely, could increase costs and result in poorer average execution price for the institutional trader who submitted the metaorder. Both types of participants are aware of each other and have beliefs – real or imagined – about the other’s behavior, which inform their own trading strategies.
Lastly, we know from the findings of order flow theorists regarding latent liquidity – and the challenges of modeling it accurately – that the orders visible in the LOB represent only a small fraction of the potential liquidity available at any given time. Liquidity can expand or contract rapidly and with little warning. Participants behave strategically, often hiding their intentions until market conditions meet specific criteria as part of their pricing procedures, or, if they already own a security, waiting for the right moment to exit. Allowing for strategic behavior (and the prediction of it by other participants) would therefore seem to be a requirement for any modeling of price formation within LOBs.
Keynesian Beauty Contest
I shall now turn to what I hope is not too radical a suggestion, given the extent to which we’ve focused on endogeneity and the importance of market participants’ expectations.
The Keynesian Beauty Contest (hereafter referred to as “KBC”) is a famous thought experiment, first formulated by John Maynard Keynes in 1936, that seeks to explain how market participants think about their peers and competitors. The metaphor illustrates how investor behavior in financial markets often depends not on fundamental valuations but on the collective perceptions of market participants. Keynes compared the stock market to a beauty contest in which participants are asked to choose the most attractive faces from a set of photographs, not based on their personal preferences, but on what they think the average opinion will be – similar to how contestants on Family Feud aren’t actually giving their own answers to a given question, but rather guessing what the most popular answers would be in a random survey sample.
In this scenario, success comes from correctly predicting the choices of others, rather than from having any objective or intrinsic understanding of beauty. In financial markets, this translates to investors making decisions not based on their assessment of a stock’s fundamental value, but on what they believe others will think about the stock’s value. This leads to a self-referential (and often irrational, insofar as it defies neoclassical conceptions of rational agent behavior) dynamic where market prices are driven more by expectations of collective sentiment than underlying economic realities. Keynes used this analogy to explain why financial markets can often exhibit volatility and bubbles, as prices become disconnected from intrinsic values and instead reflect the shifting expectations and strategies of market participants.
However, Keynes’ emphasis on deviations from fundamental value appear to be misguided, in light of the observation of strongly endogenous outcomes for order flow and consequently for price formation. Given these recent discoveries, we can reformulate the KBC such that it is understood not as a deviation from an elusive fundamental value, but as a core mechanism in the creation of market pricing – one that is intertwined, moreover, with the endogenous and feedback loop dynamics already discussed. The KBC should represent a set of beliefs market participants have about the future state of the LOB given other participant beliefs (real or imagined). To the extent that the notion of fundamental securities valuations might gain traction, it is from its utility as a proxy for the prices submitted into the LOB by those participants who take ideas of fundamental valuations seriously. At the very least, we can remain constructively agnostic as to the existence of fundamental or “correct” prices, and focus only on what was submitted into the LOB.
If we conceive of the ways market participants update their beliefs in terms of a KBC mechanism, we gain a nuanced understanding of how traders perceive and respond to the presence of strategic behavior, particularly from institutional traders executing metaorders and other market participants, such as high-frequency traders (HFTs), seeking to capitalize on these large trades. In this framework, rather than simply reacting to observable market data or their own analysis, traders would update their beliefs based on what they think others believe about the presence and strategies of key players.
For instance, institutional traders attempting to execute large metaorders with minimal market impact might operate under the radar, using tactics designed to avoid detection and price slippage during their metaorder execution. However, other market participants, aware of the potential for such camouflaged strategies, might begin to speculate about the presence of hidden metaorders. The KBC process would suggest that traders, instead of acting purely on direct signals, would base their actions on their expectations of how other traders are interpreting the market. This would involve a recursive loop of guessing and second-guessing: HFTs might adjust their strategies based on what they think other HFTs or market makers believe about the presence of a metaorder, while institutional traders might alter their tactics in anticipation of the HFTs’ moves.
This mechanism adds a layer of complexity to market behavior, where the belief formation process itself becomes a key driver of order flow – or more specifically, of new orders. Traders might become more sensitive to subtle clues in the LOB, such as changes in order flow or shifts in liquidity, interpreting them as signals of underlying strategic behavior. This could lead to self-reinforcing cycles where the mere suspicion of a metaorder prompts behavior that either brings the metaorder to light or causes price movements that would not have occurred otherwise. In this way, the KBC process would not only influence how traders perceive the actions of institutional players, but also actively shape the evolution of prices in the LOB, making the market more reactive and potentially more volatile in the presence of strategic trading behavior.
The KBC process could be integrated into the dynamic feedback loop that order flow theorists believe governs the evolution of the state space of the LOB. In other words, incorporating the KBC process into the self-exciting feedback loop of price formation under this model could more fully capture the data-generating process.
Now, regarding the KBC process itself, what are participants considering when thinking about the actions of others? If we apply the KBC process broadly across the entire LOB, the model would quickly become overly computationally intensive, not to mention difficult to calibrate empirically. To avoid this, we need to limit the dimensions of analysis – specifically, the levels of reasoning participants engage in and the types of events they form beliefs about within the KBC process. We can simplify this, on the one hand, by focusing on whether or not a metaorder is present in the LOB, and on the other, by considering the actions participants might take, such as choosing price levels within the LOB for order submission or opting for market orders if they prioritize immediacy. This approach increases the likelihood that the model will remain computationally manageable.
Experimental research into KBC processes in financial markets is limited, with most studies using more abstract setups typical of standard game theory rather than directly addressing asset markets.74 There is seemingly just one published example of a KBC experiment designed specifically for financial markets: Hirota, Kusakawa, Saijo and Tanigawa, “Keynes’s Beauty Contest in Stock Markets: An Experimental Study” (2018). Presented in 2019 at the annual derivatives market conference of the Auckland Centre for Financial Research. Keynes originally suggested that the optimal strategy in a KBC context is to guess the average opinion of other participants. What we’re attempting here – integrating a KBC process into a model for price formation in LOBs – is a good deal more difficult, because in a field of heterogeneous and competing strategies, determining just what an “average opinion” might be is far from straightforward. If it proves too complex or theoretically unsound, then so be it. However, the KBC process strikes me as a potentially elegant solution for integrating – or endogenizing – latent liquidity in modeling LOB dynamics. Ultimately, this could offer a way to model the complex process of price formation in securities markets. The concept of an underlying KBC process as the generator of agent beliefs about other future trading activity (and thus price formation) seems like a promising candidate.
To implement this model mathematically, I have developed two different formulations of the same general idea, which are detailed in a mathematical companion piece to this essay. Both formulations propose that there is an underlying structure to the belief formation of market participants, which influences their eventual order placement in the LOB. This belief formation approximates a KBC process and fits into an endogenous, continuously updating feedback loop that connects the current state of the LOB, its past states, and participant beliefs about the future.75 If you would like to take a detour into equation-town, I encourage you to read through the mathematical appendix that accompanies this essay (beginning just after its concluding page, in print edition). I’ve included derivations for the basic model setups for both formulations. Additionally, you’ll find derivations and solutions to several of the seminal market microstructure models discussed earlier in the section on the intellectual history of the subfield.
Non-Equilibrating Diffusion-Reaction Model
Recall that in Part III we reviewed the application of diffusion-reaction models, originally developed to analyze chemical processes that tend toward an equilibrium, to price formation in financial markets. One formulation of my KBC model for the arrival of metaorders into an LOB incorporates a variant of the diffusion-reaction model that operates within a non-equilibrating framework, reflecting the inherently dynamic and volatile nature of financial markets. This model should capture the complex interplay between the visible LOB, the latent metaorders, and the heterogeneous beliefs of market participants.
In this model, new orders are not generated according to an ad hoc stochastic function, but instead are a direct response to the evolving state of the LOB, which is governed by an underlying, two-dimensional KBC process. The KBC process reflects the recursive nature of belief formation among participants: each agent forms beliefs not only about the current state of the LOB, but also about the beliefs of other participants regarding the presence and impact of metaorders. This recursive process influences their expectations about future price movements within the LOB.
As metaorders begin to diffuse into the LOB, they can trigger a cascade of reactions among participants once they are identified. These reactions are modeled as a diffusion-reaction process, where the arrival of new orders prompts price revisions in existing orders and potential cancellations. However, unlike traditional models that might push the system towards equilibrium, this model is designed to remain non-equilibrating. The ongoing flux in participant beliefs, driven by the KBC process, ensures that the system never settles into a steady state. Instead, it continuously adapts as new orders enter the LOB.
The reactions of participants to the arrival of metaorders are inherently heterogeneous. Different agents (institutional traders, HFTs, and retail traders, for example) possess varying levels of risk tolerance, along with varying interpretive frameworks and methods of reasoning, leading to a diverse set of responses. Some may revise their orders aggressively, anticipating significant price changes, while others may cancel orders or place new ones to capitalize on perceived opportunities. Still others may simply wait. These varied responses contribute to the nonlinear dynamics of the LOB, further reinforcing the model’s non-equilibrating nature.
Overall, this diffusion-reaction model should provide a more nuanced expression of how metaorders influence the LOB, emphasizing the importance of participant beliefs and the recursive, self-excitational nature of those beliefs. By incorporating a KBC process and allowing for heterogeneity among agents, the model captures the complexity and relative unpredictability of real-world financial markets, where price formation is an ongoing, dynamic process rather than a movement towards equilibrium.
Multi-Dimensional Hawkes Process with
Non-Parametric KBC-Informed Price Return Distribution
Alternatively, a multi-dimensional Hawkes process76 For two recent examples of multi-dimensional Hawkes process models for LOB dynamics, seek out the following: Lu, X., & Abergel, F. (2018). High-dimensional Hawkes processes for limit order books: modelling, empirical analysis and numerical calibration. Quantitative Finance, 18(2), 249–264.https://doi.org/10.1080/14697688.2017.1403142, and Swishchuk, Anatoliy, and Aiden Huffman. 2020. “General Compound Hawkes Processes in Limit Order Books” Risks 8, no. 1: 28. https://doi.org/10.3390/risks801002. with a non-parametric KBC-informed price return distribution could provide a parsable framework for modeling the dynamics of an LOB. This model captures the ever-changing nature of market behavior, accounting for fat-tailed distributions and fluctuating liquidity. The core of the model lies in its multi-dimensionality, where multiple types of events are tracked simultaneously. In the context of an LOB, these dimensions could represent different types of orders, such as market orders, limit orders, and cancellations at various price levels. Each dimension is self-exciting, meaning an event in one dimension can trigger more events within the same dimension or across others, allowing the model to reflect the intricate web of influences within the LOB.
With the incorporation of a KBC concept, the model avoids assuming either a fixed distribution for price returns or one determined in ad hoc fashion by a stochastic function. Instead, it employs a non-parametric approach, allowing the price return distribution to be shaped by actual market data. This flexibility is crucial in capturing the fat tails often observed in financial returns, where extreme events occur more frequently than a normal distribution would suggest. The KBC-informed price return distribution adapts to real-time market dynamics, reflecting the evolving beliefs and actions of participants who are constantly trying to anticipate the behavior of others. This recursive process of belief formation adds depth to the model, making it more responsive to the complex realities of market behavior.
Unlike traditional models that assume markets naturally move toward equilibrium, this model is designed to be non-equilibrating. The self-exciting nature of the Hawkes process, combined with the feedback loops informed by KBC behavior, together model a market that is in a consistent state of flux. Significant events, such as large orders or notable price movements, can trigger chains of reactions, leading to prolonged periods of high volatility or liquidity shifts without ever settling the system into a stable state. This reflects something closer to the true nature of financial markets, where equilibrium is more of a theoretical concept than a practical reality, and where market conditions are continuously evolving in response to new information, participant behavior, participant beliefs, and beliefs about their beliefs.
The model’s ability to generate fat-tailed distributions of returns emerges from the interplay between KBC-informed branching and the non-parametric nature of the price return distribution. As market participants react not just to current prices but to their expectations of others’ reactions, extreme events become more likely, contributing to the fat tails seen in real market data. Liquidity, too, fluctuates as these dynamics unfold: periods of high activity can temporarily drain liquidity, while calmer periods see it replenished. The non-equilibrating framework of the model ensures that these dynamics are ongoing, providing a realistic and flexible representation of how markets function, accommodating the chaotic and sometimes extreme behaviors that characterize real-world trading environments.
Conclusion
If you want to learn about how the prices of securities are set, the first and best place to look is the LOB. Order flow through an LOB gives rise to market prices (definitionally, the price resulting from the last-executed trade), and these form the price history. Several theoretical models from the last century set foundational precedents for current attempts to determine security prices: Bachelier’s study of price changes using Brownian motion (informed by Leon Walras’ earlier auctioneer model), the Black-Scholes option pricing model, the Kyle and Glosten-Milgrom models that were the first properly market microstructural models of market makers. Building on these past insights but incorporating vast troves of LOB data, the contemporary order flow theorists have put financial markets under the proverbial microscope to develop microfoundations for securities market dynamics. Having passed through a period of explosive empiricism, order flow theorists are content to let the data do the talking – an extremely welcome inversion of the usual neoclassical impulse to fit data to preconceived models.
What the order flow theorists seem to have found after poring over the data and developing myriad models for the behavior of order flow over the years is a remarkable set of statistical regularities that will serve as the basis for many future studies. These empirical properties are suggestive of a fine structure, or resilience, to order flow that ironically stands in contrast to the variability and unpredictability of financial markets, which ultimately keeps participants coming back to trade in the first place. The intuition of many order flow theorists that any reasonable model for order flow (and by extension, any model for price formation) must be able to reproduce these statistical regularities – sign autocorrelation, clustering of volatility and trading activity, the square-root relationship between price impact and trade size, the self-referential, endogenous characteristics of trading behavior, and the rapid entry and exit of liquidity to and from the LOB (sometimes theorized as latent liquidity) – is certainly a powerful one. I would argue that it should serve as the basis for a productive constraint on future market research.
I’ve made my own small contribution to their laudable effort by modifying a rather old idea from economics, the KBC process, in order to develop two separate models for price formation. I’ve proposed that an underlying KBC process may be driving the belief-updating procedure of market participants in response to strategic behavior, and that the key types of participants are those seeking to trade large quantities of securities undetected (i.e., place metaorders) and those who know a metaorder may be present in the LOB but can never be entirely sure of what they’re looking at. Equipped with these assumptions, my analytic models could serve as a basis for future simulations of order flow, or even have real-world trading applications. I leave it up to readers, particularly those with an interest in financial mathematics or machine learning, to explore these possibilities further.
With all this said, we’re left with a few intriguing open questions that deserve careful consideration.
In contrast to the price-setting procedures for industrial firms, we don’t yet have comprehensive studies that bring together, in price-administrative terms, the many different pricing procedures employed by HFTs, institutional traders, and other financial market participants. We have yet to develop order flow theory’s own version of Lee’s famous appendix to his study on surveys of pricing managers. The trouble with accomplishing this (and probably the reason such a wide study generally isn’t done) could be that the pricing rules of trading firms are so much more closely guarded – precisely for the reasons we’ve been discussing in this essay. They would be giving away their current thinking regarding the evolution of the limit order book and price formation, knowledge of which could alter how other participants choose to operate, thereby forcing them to develop new pricing models in response. Still, it ought to be possible at the very least to develop a catalog of past price-setting models. These models span a breadth of setups incorporating far more than just the Black-Scholes option pricing or Almgren-Chriss optimal execution models or other similar mainstays. A survey of the full variety of models could aid in theorizing more granular price-setting behavior, including when participants decide to submit their limit orders into the LOB.
It was beyond the scope of this essay to consider the institutional development of exchanges, though they clearly play a critical role in facilitating the trading venue and supplying it with “rules of the road” for how trades can be placed and when (or if) they will be executed. A historical analysis of the development of exchanges from an institutional lens – in particular, one informed by the findings of order flow theorists, such as the historical transition of trading in many different asset classes from quote-driven marketplaces to order-driven ones – could yield meso-scale insights that complement the micro-scale statistical mechanics we’ve been investigating throughout this piece.
Lastly, my own models that I’ve presented here are intended as general analytical models attempting to integrate the idea of the KBC into existing order flow theoretical models, the better to capture the logic informing beliefs among participants. The exact mechanism by which the KBC process could slot into the overall dynamic feedback loop of price formation identified by the order flow theorists needs further exploration. For instance, several questions remain unanswered: What are the parameters of attention for such a KBC process? Could these beliefs potentially be grouped together into regimes of beliefs that could help explain the often sudden switch from a relatively sedate type of order flow (say, a steady influx of metaorders that is variously undetected or simply matched with adequate liquidity so as not to cause noteworthy price impact) over to a highly clustered volatility of trading activity? Or, what governs phase changes among clusters of beliefs?
Financial markets are best known for their wide and often unpredictable changes in price. It’s arguably why we keep coming back to them as participants. But just as noteworthy are the deep structures of order flow that give rise to those fluctuations. If we take order flow theory seriously, then it’s clear that in financial markets, too, it is people who set prices. And it is those same people who ultimately cause (intentionally or not) both the turbulence and the elusive, though resilient, fine structure we observe. ~
