The risks of backtesting: why a good track record guarantees nothing
A backtest is a simulation of what would have happened if you had applied a strategy in the past. It is a useful tool, but misleading if not interpreted properly. These are the problems that ruin portfolios most often.
1. Overfitting
If you test hundreds of rule combinations, some will come out wonderful by pure chance. A strategy with many parameters tuned to the past usually performs worse in the future.
How to detect it: few rules, "round" parameters (1, 3, 6, 12 months), and results that hold up when each parameter is changed slightly.
2. Data snooping
Even if the author does not tune on purpose, they know the past. Keller's papers are published with data up to 2022; the behaviour afterwards is the real exam, the out-of-sample period. That is why on every sheet we show how it has done since publication.
3. Survivorship bias
If you test only ETFs that exist today, you ignore those that closed. In strategies using ETFs on major indices the effect is smaller, but it exists.
4. Ignored costs
Commissions, bid-ask spread, taxes and ETF tracking take points off the return. In strategies with monthly turnover they are significant. That is why our backtests include 0.1% per trade and you have a simulator with adjustable costs.
5. Unrealistic execution assumptions
Trading at the exact closing price of the last day of the month is easy on paper, less so in real life. If you trade the next day, the results change somewhat.
6. A short or favourable period
With ETFs there is only data since 2007-2008. A period dominated by rock-bottom rates and a big bull market does not represent all environments. Some papers use data since 1970 with indices; that is more information, but with more uncertainty.
7. Regime change
What worked in a low-rate environment may not when rates change. It is exactly the reason HAA incorporated an inflation canary.
How to protect yourself
- Be wary of any backtest with returns that are "too good".
- Prefer strategies published by researchers who also show their worst periods.
- Compare with a simple passive benchmark (such as the 60/40).
- Use live tracking: go to the tracking page and check how the recorded signals evolve.
- Invest only what you can afford to hold in a strategy that may do worse than a passive one for years.
Frequently asked questions
Are backtests good for anything?
Yes: for discarding bad ideas, understanding how a strategy behaves in a crisis and estimating orders of magnitude. They are not good for promising returns.
What is the out-of-sample period?
The stretch of data after the strategy's development, which was not used to design the rules. It is the best check that there is no overfitting.
How long does it take to evaluate a tactical strategy?
Many years, and several market cycles. A few months or a year say almost nothing.