Finance strangely views randomness very differently than other academic fields.
In physics, gas molecules behave randomly. They could be moving any direction at any speed. You’re not going to predict them with much accuracy. However, when you group them together and measure the properties of the gas collectively, the behavior doesn’t seem random any longer.
The collective pressure they produce on their container is known. If the gas is expanding, its expansion rate is known. The change in temperature during expansion is also known.
Randomness at the molecular level doesn’t create randomness at the macro level. It creates order at the macro level.
Unlike physics, finance expects to see pure randomness at both levels.
Is Pure Randomness at the Macro and Micro Level Possible?
I’ve said many times a market index is not the same as stocks. They are different. A market index is a collection of stocks. Randomness in one does not look like randomness in the other. The base assumption that randomness at the single security level (often characterized as a random walk without trends) should look and behave like randomness at the market index level isn’t really logical as this isn’t how randomness works in the rest of the world.
I didn’t truly grasp the full implications of this until a 5th Century BC Greek philosopher on twitter (who now lives in Germany) opened up my eyes.
Twitter Epiphany
A year ago I posted some charts on twitter. The charts were initially constructed similarly to the game in my post on the arithmetic return.
But instead of looking at all the individual samples, I then grouped them together and looked at how the combined wealth moved through time. Importantly the wealth was not rebalanced.
Notice any similarities in these charts? They all begin heading upwards. All of them. Some longer than others, but the first move is the same and very clear. Then they peak, flatten out for a bit, and ultimately roll over. They do this in different ways. This is still random but the pattern is fairly clear.
The tweet was meant to show how portfolio compound randomness has an expected shape. We know to expect a convex, exponential shape with compound growth of a single item. But this isn’t the only type of curve random growth produces.
It’s possible to also expect a concave shape. A collection of random returns –if those random returns have a positive arithmetic return and a negative geometric return–will rise up, flatten out, and then roll over.1 This convex shape isn’t random either.
Chart for demonstration purposes. Calculations are my own.
This tweet was written soon after my post on momentum. Momentum and trend following are similar in that they suppose a stock that is already going up is more likely to keep going up and a stock that is going down is more likely to keep going down.
The shape of these purely random charts is similar to those found by momentum researchers. They say that a collection of stocks which rise up and then fall a few months later must possess a form of momentum which then fades away after time. Of course, a collection of random assets can also do this on their own, which brings into question the extent of “stock momentum”.
The Tweet That Changed Everything
And then I got a tweet that changed my view on this forever.
Is Parmenides von Elea saying to trend follow coinflips. Really? Trend follow randomness? Isn’t a key trait of randomness that it doesn’t trend?
Now I can understand trend following when “rebalancing between assets“. But there isn’t any rebalancing in this chart.
Did Parmenides lose a step or two mentally over the last few centuries? I try to be nice on twitter, but I was this close to calling Mr. von Elea a fool.
And then I looked at the charts again. Hmmm….
Trend following would have worked well. A simple moving average type system looks like it would have caught the up and down moves. Must be a coincidence.
I looked at the rest of them. Same thing. He was right.
Go back and look at them yourself.
How can this be? One chart, whatever. But multiple of them? That’s quite a big coincidence.
And then it hit me:
I was describing how randomness exhibits momentum in a collection of stocks in order to show that individual stocks don’t have momentum, totally ignoring that “randomness trends in portfolio form” was just as important.
Let’s Work Through Why
Before we dive in, it is vitally important you understand my post on the arithmetic return not existing through time.
With enough repetitions the returns you actually receive are much much closer to the geometric return than they are to the arithmetic return.
Notice in this chart, the random samples diverge from the red arithmetic return. The red line isn’t a good expectation of the future.
Instead, compound randomness moves toward the geometric return not the arithmetic return as it moves through time. You should expect the geometric return.
The Game
To explore this phenomenon, we’ll use a coin flip game of:
- heads up 50%
- Tails down 34.5%
The arithmetic average is 7.75%
The geometric average is -0.88%
The standard deviation is 42.25%
The coin portfolio is comprised of 100 coins starting with a dollar wagered on each. The outcome of the flips are re-invested in the next round on the same coin, never rebalancing between coins.
The Charts
We’re going look at two values in the chart simultaneously. The main blue line is the portfolio value (scale on left). The second orange line represents the portfolio concentration (scale on right).
The concentration shows us how the wealth is spread through the portfolio. It is the sum of the square of the portfolio weight of each coin in the portfolio.3 A value of 1 means all the wealth is in 1 coin. A value of 0.01 means it is evenly balanced between each coin.
Chart for demonstration purposes. Calculations are my own.
Let’s put a bit of math into this. I promise it will be short.
Early on the wealth is balanced over 100 coins. So each coin’s weight is 1/100th of the portfolio. The concentration value is
0.012 x 100 = 0.01.
For our coinflips portfolio the standard deviation (also called volatility) is:
This means the beginning standard deviation of the portfolio is really, really low. It’s practically zero.
The geometric return equals the arithmetic return minus half the standard deviation squared.
Arithmetic Average – Volatility2 / 2 = Geometric Average
So at the start the portfolio looks like this:
7.75 – (.01*42%^2) / 2 = 7.66%
The main thing to remember: low concentration means low standard deviation and high geometric return. High concentration means high standard deviation and low geometric return.
At The Start
With a low standard deviation, the geometric return is very nearly equal to the arithmetic return at the start. The arithmetic return is quite positive. With both the portfolio arithmetic return and the geometric positive with very little volatility, the portfolio value naturally shoots upwards.
Chart for demonstration purposes. Calculations are my own.
Flattens out
But over time the wealth does not stay balanced. The portfolio changes. It evolves. Returns will spread out and wealth will concentrate. As it spreads out, the portfolio concentration starts to rise. The standard deviation of the portfolio grows with it. Now there is a bit more volatility drag. Therefore, the Geometric Return isn’t as high and nears zero. The portfolio growth flattens out.
Chart for demonstration purposes. Calculations are my own.
And Then Rolls Over
At some point, the wealth concentrates into fewer and fewer assets to the point that the volatility drag of our less balanced portfolio overwhelms the arithmetic return. The portfolio trajectory peaks and rolls over. The paths turns downwards.
This is exactly what I talked about with randomness in the momentum portfolio. We’re just looking at it now from the opposite perspective. From the portfolio perspective.
Chart for demonstration purposes. Calculations are my own.
It’s a choppy path downwards, but clearly downwards. And it will now head that way, in general, forever.
The Portfolio Trends
So the overall shape here creates a very interesting realization. Early in time the portfolio is nearly guaranteed to go up. Later in time it’s likely to be flat, and at the end it’s likely to be sloped downwards.
This smells of non-randomness, right? Randomness doesn’t expectedly change direction through time. But of course this is coin flips; it’s all randomness.
Chart for demonstration purposes. Calculations are my own.
At a high level, the chart looks like it’s autocorrelated, aka, the prior return influences the next return, aka, it trends. If the trend has been up, it will likely stay up, and if the trend is down it will likely stay down.
However any test of autocorrelation from one period to the other will fail every time. It is just random coins. The arithmetic return of each period is still slightly positive and never changes. There isn’t any auto-correlation from one period to another.
But if you group periods together so that the autocorrelation sample takes the results of compounding over multiple rounds, say after every 100 rounds, the entire equation changes. Now we’re in average compound return space, and in the space where the arithmetic return doesn’t exist.
You’re going to “see” the geometric return here, and that is much more likely to “trend” because the weight and distribution of the portfolio are likely to stay in a similar state over the near term, but won’t stay that way forever.
Importantly, you would never see this autocorrelation in any of the individual coins at any time period. But you can see these autocorrelation effects in a portfolio of coins.
A portfolio of random coins can therefore trend. They gain a bit of order by grouping them together.
But There is More…
Now notice this is still a random return stream. It’s not a smooth ride down, especially since the portfolio is fairly concentrated at this point. Most would look at these bounces up and down and view them as purely random, but are they?
Chart for demonstration purposes. Calculations are my own.
Some of them are, but it looks like after the biggest falls the portfolio always rebounds. Can that be right?
Well for large parts of the downtrend the portfolio is concentrated into essentially a single coin. Not exactly a single coin, as there are still some coins down at lower wealth levels. But the wealth of one coin dominates the portfolio. This is why the orange line tracking the portfolio balance is often nearly 1.
That is until that one coin goes on a losing streak.
Chart for demonstration purposes. Calculations are my own.
When the primary coin hits a bunch of tails, its value plummets. As its value plummets, so does our portfolio since that coin is the core part of the portfolio. But notice when the value plummets what happens to the portfolio concentration.
It spikes down! Our lead coin has fallen back in line with a few other coins, and now our concentration isn’t so poor any longer. The portfolio’s fall therefore pushes the geometric return back into a positive position. Now the portfolio should trend back up again for a bit, or at a minimum stop falling.
So after the big crash the rebound wasn’t random. It was expected because the geometric return flipped back to positive again.
Once again you will never see this on a single period scale. It is entirely a function of everything trending towards the geometric return and looking at it from multiple time periods.
Spikes Up Then Down
The opposite is also true. When you see a big spike up around the middle part, its likely because a single coin went on a run of heads. This run of heads threw the portfolio further out of balance.4
Chart for demonstration purposes. Calculations are my own.
Now, after that run of heads, the geometric return flips from positive to negative. Therefore the portfolio should be expected to roll over and fall from its high.
There is Also Mean Reversion!
When the portfolio spikes down, the improved geometric return causes it to revert back upwards. When a portfolio spikes up, the degraded geometric return causes it to fall back down. The coin portfolio looks like it mean reverts.
In a prior post, I pointed out how strange it was that studies found evidence of both trend following and mean reversion at the same time within the same market index. Yet here we are, a portfolio of coinflips which looks like it trends, and mean reverts at the same time. Interesting….
It is All Fractal
So now we see, there is a master, overriding shape. But even within that shape the pattern repeats internally. There are crashes down which re-set balance and a positive geometric return. Followed by gains that ultimately dissipate as the portfolio becomes more concentrated again, until rolling over to continue its decline.
The whole process is fractal. The curved shape repeats over and over again at smaller scale.
Chart for demonstration purposes. Calculations are my own.
Notice how these fractal curves in the coin portfolio (green) match with curves in the portfolio concentration (black).
The rallies start when the portfolio has some balance and the geometric return is positive. As the rally runs, the balance goes away. It peaks, often in a spike, concentrated with all balance gone. Now concentrated–and with a negative geometric return–the portfolio reverts and heads downward. As it falls downward, the portfolio balance comes back, flipping the trend and starting the process over again.
Portfolio Volatility is the Ship’s Rudder
The coin portfolio doesn’t grow at the arithmetic return, it grows at the geometric return.
So when portfolio volatility is low, the portfolio will grow quickly, When portfolio volatility is high, it will grow slowly. When the coin portfolio balance crosses a threshold, the portfolio’s geometric return flips from positive to negative, and vise versa.
This is why the portfolio curves. The charts show the portfolio degradation from balanced to concentrated which moves the volatility of the portfolio, and ultimately the geometric return over time.
Therefore the portfolio volatility steers the direction of our coin portfolio like the rudder on a ship. This behavior causes the coin portfolio to trend and mean revert.
Trend
The portfolio geometric return changes through time because the portfolio volatility changes through time. So even though none of the coins “trend” the portfolio can, will, and should trend. It will trend up when when the geometric return is high, and down when it is low because the portfolio balance usually doesn’t change quickly.
Linking to trend following: Seeing a strong return over a few periods (not one step), makes it likely that the portfolio geometric return is also positive. Seeing poor returns over a few periods makes it likely the geometric return is negative. The trend should persist because these conditions within the coin portfolio should continue in the short term.
Mean Revert
Once the opening burst is over and the portfolio weights become somewhat concentrated, the portfolio becomes susceptible to mean reversion. Tops in the coin flip game are usually when the prior winners have a long winning streak, further concentrating the wealth. But this concentration creates a negative geometric return, leading losses soon after. AKA, big portfolio wins are likely followed by losses.
Same is true on the bottom. The big falls often come from the single concentrated portfolio coins having a bad streak. However this bad streak means the coin portfolio is no longer as concentrated. So the geometric return flips from negative to positive. Therefore big portfolio losses are likely to be followed by wins.
Application to The Stock Market
Now the real market is obviously more complicated than a coin flip portfolio.
My main goal with this post is to show that just as a collection of molecules don’t act so random at the macro level, most collections of randomness don’t act as random at the macro level. A portfolio of random coin flips will look “less random” than the individual coin flips themselves. So why should we expect any different from stocks and a market index?
We must question the foundational assumption of what makes a market efficient at the macro and micro level.
The prevailing belief that a market index should look totally random is maybe not reasonable. Is it even mathematically possible for the market to behave randomly at both the stock level and the market level? Maybe not.
But all this coin flip theory does us no good unless we try and map it into the real world.
Concentrated Indices Spell Trouble
The easiest conclusion is that a concentrated market index is potentially trouble. And there is some historical data which backs this up. Entering 2000 the S&P 500 was historically concentrated. We know how that played out. Also historically when the “market breadth”–the number of stocks which rise during a rally–is low, returns usually lag soon after. This is very similar to when a single coin peaks up before reverting back down.
I’m sorry to say, we are in a similar market today.
Now obviously the market doesn’t concentrate to the level our coins did. So this observation is mostly just conceptual. But it leads us to the next observation.
Correlation Moves the Rudder in the Real World
The biggest difference between coins and the market is the existence of correlation in markets. Coin flips are always uncorrelated. Investments are not.
Correlation plays an enormous role in portfolio standard deviation.5 In some ways it is more important than the individual stock volatilities. In investing correlations change.
So in the real world while portfolios will not degrade and become fully concentrated like our coin flip example, they are very likely to experience portfolio standard deviation changes through shifts in correlation. Therefore I believe it is critical to monitor portfolio correlation. Essentially correlation is the rudder on the investment portfolio ship, charting its most likely heading.
An Intellectual Support For Trend Following?
Investing may feel like one step. But it is not. A month is a combination of many days. A day is actually the combination of many hours, which is the combination of minutes, which is a combination of many seconds. All data we get in investing has been compounded over multiple steps. Therefore our investing experience is likely to track to the geometric return.
The blogger Jesse Livermore in his post on trend following stated6:
The [trend following] backtest will reveal an unexpected result: that the [trend following] strategy works very well on aggregate indices–e.g., the S&P 500, the FTSE, the Nikkei, etc.–but works very poorly on individual securities.
I still believe individual securities are fairly random and don’t trend.7 I believe the data mostly backs this up. But indices (portfolio’s of stocks) can trend. I’ve shown here that relationship between a portfolio and its components isn’t really unusual.
Even if the underlying stocks behave randomly, it seems trend following on an index may have mathematical merit. If an index is trending up or down, then it is likely its correlation and internal weighting support that trend. If neither of those change, then the “trend” of the portfolio should probably continue.
Now to me, I wouldn’t trend follow based solely on price alone. I think it makes more sense to monitor standard deviation and internal correlation of the index, try and derive where the index’s geometric return is in its process, and then bail on the trend when the standard deviation, correlation and price say the trend is over.
I haven’t executed these ideas yet, but I’m currently studying how to best implement them.8
Is Trend Following All Hot Air?
Natural process often show randomness at the micro level and order at the macro level. In physics, gas molecules are a random mess, but the properties of the entire gas show order. Their collective path through time as they expand, contract, and move is predictable.
In thermodynamics, the temperature of a molecule is hard to pin down, but the temperature of the object and the rate of temperature change through time of that object are predictable.
Hot air is predictable, even though the properties of a single molecule are not.
I’ve shown here that compounded coin flips–the embodiment of randomness–don’t act so random when collected together. Their returns through time certainly don’t look random as they trend and mean revert. They have some order. Yet this predictable motion is created by randomness. Coin flip portfolios behave like hot air.
Stock markets behave essentially the same way. Random at the stock level, less random at the index level with signs of trends and mean reversion.
Which begs the question: do index trends and reversions indicate market inefficiency and behavioral bias, or instead reveal that many investing theories are full of hot air?
Footnotes:
1-The shape of moving from the arithmetic return to the geometric return is the same with any set of returns. But a positive arithmetic return and a negative for flat geometric return make this effect most noticeable visually because they create a concave chart.
3-These may seem a little strange, but I went with this definition for portfolio weight as it ties into the formula for portfolio standard deviation.
Now if each coin has the same standard deviation, then the standard deviation of the portfolio is dependent on the weight or balance of the portfolio. As I explain later in the post, that means this value solely determines the standard deviation of the portfolio at a certain time, and therefore, also determines the portfolio’s geometric return.
4-It is harder to see here as the scale isn’t the best, but the portfolio is being driven closer and closer towards 1 even though it looks like a flat line. Once the line looks flat like this it’s only luck keeping the portfolio from heading downwards again.
5-In the Geometric Frontier post I showed how to calculate the standard deviation of three assets. Notice the correlation (p)
shows up 3 times. You would see it 4950 times in a 100 asset portfolio.
6-He answered this problem differently than me, but it’s still an enlightening answer and well worth your time. Interestingly his answer also boils down to volatility.
7-If you notice in the trend following post I specifically stated trend following on indices is different.
8-Its not as simple as it seems since stocks are more complex than coins.
This suggests support for 1/n strategies?
No, not necessarily, although it certainly doesn’t suggest against those strategies.
Hi, I was hoping to recreate your visualizations to gain a better understanding. However, I’m struggling to make those average wealth charts. I have a dataframe, each row is a coin, and each column is a point in time. I took the log10() of it to plot the twitter photo just fine. Now when I take the mean() of the dataframe and plot it I get almost a line that descends with a -.4 slope(it isn’t perfectly straight, I can see variation from random behavior). My point is that I don’t get any ascending behavior before it descends. Also, the random walk behavior is extremely small. It seems to almost cancel out in the summation. Is there a part of your analysis that I missed?
Thank you!
I’m using:
100 coins
25000 throws
heads payoff = 2 * asset value at t-1
tails payoff = (1-.51) * asset value at t-1
did you take the log10 before or after averaging the columns?
Does this mean that the index return could be attributed to 5-10 of the highest weight market cap stocks then? Whilst the SP500 contains >500 ‘gas molecules’ could 5-10 of those molecules drive 90% of the index performance? I’m thinking here of the highest Market Cap weighted stocks for the given period.
I saw some interesting Hurst Component work recently that backed-up your claims. The study was conducted over many cycles and showed that the SP500 (in this case the $SPY ETF) mean reverts in the short run, delivers Brownian Motion in the mid term and trends over the long term.
Interesting about the study. What was it called if you can remember?
I think at times the index return is attributed to the top 5-10 stocks. I think tops will occur in these markets, but that market doesn’t mean the top is near.
So how well would you expect S&P500 to do versus S&P5 (five)? Especially if you equal weight the latter?
The S&P 500 generally seem to behave like about 25 equally weighted stocks at the same internal correlations as normal stocks, So I think the S&P 500 would normally do better.
Is it possible you confuse “random” with “chaotic”? What you describe sounds chaotic to me…I am no authority, yet the distinction is significant. It doesn’t take anything from the interesting idea of underlying geometric…thanks
I don’t think so for the coin flips. The coin flips are random. Now for actual markets, I’m very open to the idea they are more chaotic than random, but its often very hard to tell the difference.
One of the things with trend following approaches is that the market adjusts to arbitrage them out. This is a real issue with using older data for backfitting. The market doesn’t behave like it used to not too long ago.
For example, I came up with a fairly simple approach using VIX to predict ahead a week or two; this was wonderfully predictive from 1993 until September 2012, and would have returned 46% CAGR on the S&P for the 20 years. The same general approach worked with S&P, NASDAQ, emerging markets, real estate. Until it just stopped working at all.
It seems to me that the market behavior has become much less structured in the last 8 years, quite possibly with all the algorithmic trading.
Very interesting post. Great actually – made me think quite a bit. I think the main point is really the ‘vol drag’ you want to reduce in order to get closes to your geometric mean.
I do have a question that on the back of the Law of Large numbers, we should expect convergence to the arithmetic mean of each individual toss and thus if we did just addition rather than compounding the wealth, we should expect a rather linear ladder right?
The issue is when you play continuously compounded there is the gambler’s ruin case… in which we go into the case of Kelly fractional betting, how would those charts change if you were betting a Kelly fraction rather than 100% of the time?
Not really. The only way to “get addition” is to not invest any profit, and also to pull in cash on the side after losses to make sure you always invest the same amount. So, #1, if you lose enough and all the cash kept on the side is gone, then your are playing a compounding game again, and #2, If you do win over time, because you don’t reinvest the profits, while the absolute gains each round are arithmetic, the percent gains of the entire portfolio get smaller and smaller, so that your wealth really isn’t growing that quickly.
Therefore in a way you don’t really get a choice of which game to play. If you’re trying to grow wealth, its going to be a compounding game.
“The geometric return equals the arithmetic return minus the standard deviation squared.”
When people say an approximation is an equality, without acknowledgment of the approximation at all, trips my bullshit detector. Especially given that I know that approximation assumes a normal distribution, and we’re talking about something which is famously not normally distributed.
Please at least link back to a post where you justify this, or at least reference the citation in Markowitz’s book that gives credence to it.
Also you dropped a “half” in that sentence.
Technically the geometric return of a normal distribution doesn’t exist because there is some chance of a negative return. That equation is similar to the one for a lognormal distribution, but even that’s not exact. The difference though is very minor when the return is near zero. When the return gets away from zero, say 20%, equation gets off. This is why daily and weekly data still work well because thier expected returns are practically zero.
So how well would S&P do versus S&P 5(five)? Especially if the latter were equal weighted?
Hi BTM,
I tried to operationalize this by setting up a scanner in thinkorswim to find tickers with a positive arithmetic mean and negative geometric mean. What I found was that this basically never happens in actual stocks. Stdev is just too low. I tried several different approaches for calculating arithmetic mean here (averaging historical returns for different periods), but same result.
Any thoughts on this?