Why backtests lie

M2.6 produced a strategy with a Sharpe of 1.39, and then charged it realistic costs and watched it die. That was the honest failure mode. This lesson is about the dishonest ones — the ways a backtest reports profit that was never available to anybody.

The framing that matters: a backtest is not an experiment, it is a measurement whose bias you control. Every degree of freedom you hold — which data, which period, which parameters, which of your many attempts to report — pushes the result one way. Nobody accidentally overfits downwards.

The bugs you can actually fix

Lookahead bias. Using information at time t that wasn't available until later. It is almost never as obvious as reading tomorrow's price:

   full-sample z-scores        standardising with a mean and σ computed
                               over the whole history — the M2.6 code
                               used a ROLLING window precisely to avoid it
   restated fundamentals       today's database has the corrected figures;
                               the market traded the originals for months
   point-in-time index members yesterday's index constituents as known
                               TODAY, not as known then
   a close-price signal        traded at the same close. You had it when?

Survivorship bias. Backtesting today's index members over ten years tests a portfolio selected for having survived. The failures were deleted from the universe and from the record. This alone has been enough to manufacture entire published anomalies.

Costs assumed away. M2.6 showed a Sharpe going from 1.39 to 0.17 on the cost line alone. If a backtest reports gross returns, it has not reported anything.

These are all bugs: fixable with discipline and point-in-time data. The next one is not a bug, and it is the one that gets serious people.

Multiple testing, quantified

M2.1 established that a Sharpe estimate is noisy — standard error ≈ √(252/n), so about 0.45 on five years of daily data. Now consider what happens when you don't test one strategy, but search.

Simulate strategies with exactly zero edge — pure noise, no signal whatsoever — and report the best one found:

   5 years of daily data.  TRUE Sharpe of every strategy = 0.
   (standard error of a single Sharpe estimate = 0.45)

   strategies tried    best in-sample Sharpe    SE·√(2 ln N)
   ─────────────────────────────────────────────────────────
              1               −0.03                 —
             10                0.69                0.96
            100                1.13                1.36
          1,000                1.43                1.66
         10,000                1.74                1.92

Read the 1,000 row. A best-of-1,000 search over worthless strategies produces a Sharpe of 1.43 — the number M2.1 called "a good systematic strategy, genuinely hard." It is entirely an artefact of taking a maximum over many noisy estimates, and the growth is √(2 ln N), so it is slow but relentless: you cannot outrun it by being clever, only by testing less.

And note how easy it is to reach 1,000 without feeling like you searched:

   2 entry thresholds × 3 exit rules × 4 lookback windows
     × 5 universes × 3 holding periods × 3 date ranges  =  1,080

Nobody experiences that as running a thousand experiments. It feels like tuning. But the statistics only count how many results you looked at before choosing one, not how virtuous you felt while looking.

This is why out-of-sample testing is not optional and is also easily faked — testing on your holdout, adjusting, and testing again converts the holdout into training data. A holdout survives exactly one look.

The formal repair is to deflate the Sharpe by how many trials produced it — Bailey and López de Prado's deflated Sharpe ratio does precisely this, and it requires an input almost nobody records: the true number of variants tried. The practical repair is to write that number down as you go.

Capacity, and the profit that isn't there

Even a real edge may not be harvestable at your size. M1.5's square-root law says cost per share grows with participation, so total cost scales like Q^1.5 while gross profit scales like Q. They cross:

   gross alpha  ∝  Q            ────╮
   cost         ∝  Q^1.5        ────┴──▶  net profit peaks, then falls

   beyond the peak, trading MORE makes LESS

A backtest run on notional you will never deploy is measuring a strategy that doesn't exist at your size. This is also why genuinely good strategies get quietly closed to new money rather than scaled — a fact that looks like modesty and is arithmetic.

Regime, and the deepest problem

M2.6's final caveat was that its data came from a model guaranteeing the spread reverts, so the strategy could not fail the way real pairs trades fail. Real backtests have a subtler version of the same defect: the past is one sample path from a world whose rules were themselves changing.

Decimalisation, Reg NMS, the rise of HFT, quantitative easing, the 2020 retail wave — each altered the microstructure the strategy trades in. A backtest spanning them assumes a stationarity that observably did not hold. No statistical technique repairs this; only awareness and shorter memory do.

What honest practice looks like

None of the defences are clever. They are all structural, which is why they are so widely skipped.

   ✓  Hold out the last 20–30% of history and DO NOT LOOK until the end.
      One look. If you adjust after looking, it is training data now.

   ✓  Count every variant tried, including the ones you abandoned.
      Report it. Deflate by it.

   ✓  Subtract costs before you are allowed to be pleased. Use M1.5's
      square-root law at your intended size, not a flat basis-point fee.

   ✓  Prefer few parameters and economic reasoning. A strategy you can
      explain in one sentence has fewer ways to be an artefact.

   ✓  Paper trade, then trade small. Live results are the only sample
      you did not fit to.

   ✓  Expect roughly HALF the backtested Sharpe in live trading, and be
      unsurprised when it is worse.

That last line is the industry's rule of thumb, and everything in this lesson explains why it is not pessimism. Between multiple testing, cost optimism, capacity, and regime drift, a 50% haircut is a reasonable prior, not a cynical one.

Where M2 leaves you

You can now take a claim of edge apart: what is the signal, is it alpha or repackaged beta, how noisy is the estimate, how many variants were tried, what does it cost to trade, and how much size does it hold? That is the whole of M2.1 through M2.7, and it is a more useful skill than any individual strategy.

One assumption has been quietly load-bearing throughout, and it is wrong. Everything here — the Sharpe ratio, the standard errors, Black–Scholes, the risk calculations — assumes returns are approximately normally distributed. They are not, and the ways they fail are not small corrections. M3 is where that assumption gets taken apart.

Source: Bailey, Borwein, López de Prado & Zhu, “Pseudo-Mathematics and Financial Charlatanism” (2014) — the multiple-testing problem stated bluntly, and the source of the deflated Sharpe ratio. Campbell Harvey & Yan Liu, “Backtesting” (2015) and their “…and the Cross-Section of Expected Returns” apply the same lens to the published factor literature, with uncomfortable results. Marcos López de Prado, Advances in Financial Machine Learning, ch.11–12 for the practitioner’s version.