Lies, Damn Lies, and Investment Statistics

Investment statistics often lie. Sometimes they lie purposefully. Sometimes they lie unintentionally. I’ve looked at lots of investment statistics through the years, and I’ve come to distrust many published numbers. It’s too easy to game them. Even when honest, they don’t cleanly compare between strategies. Either way, to successfully evaluate an investment, you need to know how investment statistics can distort reality.

So let’s dive into how statistics can paint the wrong picture. For each statistic, I’m going provide examples with data for the S&P 500. I want you to look at these statistics, and imagine they are different strategies, even though the statistics I’m going to show are from the exact same return stream. But first a question to contemplate as we work through some examples of investment statistics deception.

Which Investment is Better?

Imagine you’re evaluating two investments. One of the metrics reported by both is the Sortino Ratio. Sortino Ratio is a ratio of an investment’s return divided by it’s downside deviation. It’s similar to the Sharpe Ratio, but fancier. Instead of dividing by the standard deviation, the statistic divides by the deviation of only the negative returns. This way, the ratio isn’t penalized for large upside moves.

The Sortino Ratios for the investments are:

• Investment A reports a Sortino Ratio of 1.9%.
• Investment B reports a Sortino Ratio of 1.3%.

Do you have a preference between the two? A 46% higher Sortino Ratio is a pretty large difference. If investment A charges a slightly higher fee, is the higher ratio enough to warrant the extra cost?

Ok, now onto the investments statistics, starting with return.

Return

I pointed out the issues with returns early on in this blog. There are two kinds of returns, arithmetic return and geometric return (often called compound annual growth rate or CAGR). Sometimes it’s obvious which return is reported, and sometimes it’s not.

The problem is the arithmetic return is always higher than the geometric return. Therefore, I’m sure you can see that someone trying to convince you of the greatness of thier fund would rather show you the arithmetic return because it’s always higher. However the geometric return is a better number to describe how an investment actually performs in the long run.¹

So pay attention to what return is being reported. If it’s not clear, don’t just assume geometric return is listed, because it might not be.

Maximum Drawdown

I’ve touched on this one too here. I really like the maximum drawdown statistic. It can tell you a lot about the risks in a strategy, and maybe more importantly what kind of stress your psyche could go through while invested. But comparing this statistic from one strategy to another is dangerous.

Time

The first challenge comes from differences in strategy timeframes. Now it’s a risky idea to compare any two statistics with different time histories. As an example, returns over the last 10 years could be very different than over the last 20 years. This is fairly obvious to most people.

However, maximum drawdown is different because the statistic can only get worse with a longer investment duration, whereas other statistics could get better or worse with more data.

The longer back the strategy goes, the more likely it is to have a deep drawdown. Shorter periods are more likely to have shallow drawdowns.

Look at the S&P 500 going back 10 years. A 34% peak to trough drawdown. But back 20 years, and it’s 55%. Back 100 years, it’s nearing 90%.

A longer historical period leads to a likely higher maximum drawdown. So you really can’t compare two strategies well over different length periods. But, because drawdowns can only get worse, you can learn something from two strategies that have similar maximum drawdowns over different timeframes. It’s very likely that the strategy with the shorter history is riskier.

But interestingly even when the periods lengths match there can still be problems comparing drawdowns.

Data Resolution

Some max drawdowns are quoted with daily data, some are quoted with monthly data, and I’ve even seen some quoted with quarterly data. Which one you choose makes a major difference in the statistic. Monthly data will always show smaller drawdowns than daily data and quarterly data will show even smaller drawdowns.

The higher the “resolution” the higher the drawdown. Always. This means you can’t really compare the drawdowns of two strategies reported with two different resolutions either.

Once again though, if two strategies have the same drawdown, and same period, but one uses daily data, and one uses monthly data, you can logically assume that the one which reports with daily data likely provides a smoother ride.

MAR Ratio

The MAR ratio may be the most under loved investment statistic. But this may be a good thing, as it’s very easy to distort.

The MAR ratio is a comparison of the strategy’s compound growth rate to its maximum drawdown. Simple and clean. But, as we saw before the maximum drawdown can be misleading sometimes. This means the MAR ratio can also be gamed fairly easily by toying with the historical period and the data resolution, especially since the max drawdown is in the denominator. So while I do love this ratio, be very careful with it.

Those are pretty big differences. Even a comparison between monthly and daily data implies a huge, meaningful difference, when there isn’t one.

Calmar

More historical data can dampen this effect in the MAR ratio somewhat, so if the strategy goes back a long time, you don’t have to be as concerned about this effect. However, some don’t like how the MAR ratio can punish bad performances from decades ago, and instead use the Calmar ratio.

The Calmar ratio is very similar to the MAR ratio in that it is a ratio of compound growth rate divided by drawdown, but the Calmar ratio only uses recent data, not the full life of the investment. This shorter period, often just 3 years, means the Calmar ratio is even more likely to spit out wildly different numbers with different resolutions of data than the MAR ratio is. So be very careful with Calmar ratios.

Volatility

Volatility shows the same data resolution issues that drawdown does. Even after annualizing, the metrics differ noticeably depending on the resolution of the original data.

That’s a pretty large difference between daily data and monthly data. And this difference matters a lot when you start using volatility in ratios.

Sharpe Ratio

Ah, Sharpe ratio. Many attack this oft quoted ratio as not well suited for examining investment quality. I’m not here to highlight the flaws others see in the Sharpe ratio. I’m here to show how comparing Sharpe ratios is rife with complications.

Sharpe Ratio Definition

The academic definition of the Sharpe ratio is:

( Arithmetic Return – Risk Free Rate ) / Standard Deviation

But this equation is sometimes not followed precisely.

Dropping the Risk Free Rate

One difference I often see is dividing arithmetic return by the standard deviation without subtracting the risk free rate.

Now this is logical, as it then becomes simply return vs risk. A nice clean ratio that has value in its own right. But notice, if you do this, you get a very different Sharpe ratio:

Not including the risk free rate increase the value quite a bit. The problem is, it’s not always common for literature to point out whether the risk free rate was included or not. Most of the time, the Sharpe ratio seems to be reported with the risk free rate subtracted, but not always. Therefore, you often have to try and re-create the ratio yourself to uncover if the risk free ratio was left out of the calculation.

Over recent years, this doesn’t matter much, as the risk free rate has been near zero in the last decade. But when looking at numbers back into time when rates were far above zero, including the risk free rate clearly matters a lot.

Using the Geometric Return

Sometimes you will find the strategy replaced the arithmetic return in the calculation with the geometric return. I like this change. I think it’s a great ratio, and wish it was reported more often. But it’s not the academic calculation for the ratio.

Once again, a meaningful difference. And similarly, it’s not common for the fund to spell out which is being reported. You usually have to re-create the ratio to figure out which one is provided.

Now this change is not abused like others can be abused, because replacing the arithmetic return with the geometric return in the equation lowers the reported ratio. So anyone reporting a geometric Sharpe ratio is actually under-reporting the quality of thier returns.²

Annualizing the Ratio

Now thankfully the Sharpe ratio is almost always reported as an annualized value. This is good because you can compare similarly scaled data. However, there can be differences in how the data is scaled.

When you calculate the Sharpe ratio with daily data and don’t scale it, you get this:

This value is very different than the ones shown above. So it’s clearly useless to compare Sharpe ratios calculated from different resolution data unless they are annualized. So how is the data annualized?

Well the simplest method is to multiply the higher resolution ratio by the square root of the number of periods in a year. So √(252) for daily data √(52) for weekly, and √(12) for monthly.³ Ratio first, then annualize.

But you could also convert the daily returns to an annual return, and convert the daily standard deviation to annual standard deviation and then ratio them. Annualize first and then ratio.

This is what happens when you do each:

Notice that this isn’t the same outcome. Annualizing first and then taking the ratio always produces higher results (assuming the strategy makes money). Now it’s not going to be an overly large difference when returns are low, but when the returns are higher, it matters. And once again, it’s very rare to see someone document which method they used.

Historical Period and Data Resolution

As with drawdown and volatility, the resolution of the data can greatly change the Sharpe ratio. Over the last eleven years, there are large differences in Sharpe ratio from different resolutions.

Nearly a 20% higher Sharpe ratio with monthly data than daily data. A much higher discrepancy when comparing annual data to daily data. But as the historic period increases, these ratios converge.

Over longer periods, the the resolution doesn’t matter as much (but it still matters some). Over shorter periods though, the resolution matters significantly. So you have to be careful comparing two Sharpe ratios when one strategy uses monthly data and the other uses daily data. Over time, this discrepancy disappears, but over shorter timeframes, certainly less than 20 years, you can see very different results.

Combine Issues

I’d love to say that you only ever see one of these issues with Sharpe ratio at the same time, but sometimes you can see these differences combined together. If they combine in certain ways, the differences can be stark.

The exact same return stream shows twice the Sharpe ratio over the last 30 years, simply depending on how you calculate the ratio.

Sortino Ratio

Sortino ratio is very similar to Sharpe ratio, therefore it suffers from nearly all the same problems as the Sharpe ratio. However, I want to point out the resolution of the data can distort this ratio even further.

A 10 year Sortio ratio of 1.9 (calculated with monthly data) looks very, very different than a 10 Sortino ratio of 1.3 (calculated with daily data,) even though it’s the same exact strategy. So it’s often impossible to compare Sortino ratios of two strategies, unless the historic period and the data resolution are the same.

This is the example I used to create the opening question. Both investment A and B are calculated from the S&P 500, one is just calculated with daily data and the other is calculated with monthly data. Same investment, very different statistical outcome.

So did you get tricked into thinking it was worth paying more fees for the same returns? Better here than with your own money.

Be Wary of Investment Statistics

Mark Twain famously said: “There are three kinds of lies: lies, damned lies, and statistics”. Statistics often don’t tell the truth. I hope you can see now that investment statistics are just like any other statistic: they don’t necessarily give you the true picture of reality.

Sometimes statistics lie because they are meant to lie.⁴ Other times, statistics lie even when they are presented with the best intentions.⁵

So be careful when comparing investments to each other. Don’t let the numbers “lie” to you. Look for difference in timeframes, calculations methods, and data resolution to try and uncover the true nature of the investment.

Finally, let’s not forget the most important trait of investment statistics, which is clearly stated on every piece of investment literature, but often glossed over an ignored:

Past performance is not indicative of future results.

Footnotes

1-There are niche strategies that provide value in how they interact with your core portfolio where you might care about the arithmetic return, and less about the geometric return.

2-If I see a fund report Sharpe with the geometric return and not the arithmetic return, I often gain respect for them. I feel that funds that do this often have a better understanding and focus on the power of the geometric return. Sadly, if you don’t catch this adjustment, you might under-appreciate thier performance.

3-252 days for the number of trading days in a year.

4-One of my favorite examples of a potential trick to fool an investor with statistics is outlined in Nick Maggiulli’s post on the McRib Effect (the McRib is out now, so buckle up). Great investment results may just be pure randomness. Something is going to get lucky, and you better believe that those with something to sell will let everyone know about it.

5-There are few key things I look for to try and gauge how “honest” I think the investments statistics are.

I look for what’s missing and ask why. Volatility isn’t reported, why? No Sharpe ratio in strategy that’s appropriate for Sharpe, why didn’t they report it? Many statistics provided for the last 5 years, but only total return for the life of the fund? Why?

Similarly, are standard statistics like Sharpe ratio not included, but fancier statistics like information ratio are included? There may be a logical reason for this, but they also may just be throwing out a complicated ratio that’s harder to understand and compare instead of a number that is much simpler and more common.

Look for data resolution consistency. See if they change “data resolution” between statistics. Do they report monthly returns, but quarterly drawdown? If they clearly have the data at a higher resolution, but don’t report it, they might be hiding something.

I really like it when a fund reports each and every monthly return. Then I can create my own statistics and I know that they aren’t tying to hide something from those who really care. There may still be games played in the summary statistics, but you can find them if they are.

Finally, don’t look down on an investment just because they don’t provide super high data resolution information. I always prefer higher resolutions, daily if possible, but there are plenty of strategies out there where anything less than monthly isn’t possible (fund of funds as an example). So an investment that reports daily data isn’t necessary more forthright than one reporting monthly data.