Statistical vs practical significance

M3 has been a catalogue of ways the Gaussian assumption fails. The natural conclusion — and the one I expected to write — is that the statistical machinery built on it must fail too, so your t-statistics are unreliable and your significance tests are lying.

I tested it. It isn't true, and the reason it isn't is more useful than the claim would have been.

The test that doesn't break

Simulate data with a true mean of exactly zero, run an ordinary two-sided t-test at a nominal 5% level, and count how often it wrongly reports significance. Do it for progressively fatter-tailed distributions:

   false-positive rate of a nominal 5% t-test  (true mean = 0)

   distribution      n=20     n=60    n=252   n=1260
   ─────────────────────────────────────────────────────
   Gaussian          6.1%     5.5%     5.1%     5.0%
   t(6)              6.2%     5.2%     5.1%     5.3%
   t(4)              6.3%     5.4%     4.8%     5.3%
   t(3)              5.8%     5.0%     5.1%     5.0%
   t(2.5)            5.2%     4.5%     4.2%     4.5%
   t(2)  ← infinite  5.1%     4.4%     4.3%     4.4%
          variance

The rate sits at roughly 5% everywhere. Even t(2), which has infinite variance — a distribution so heavy that the Central Limit Theorem in its standard form does not apply — produces a conservative 4.3%, not an inflated one.

The reason is worth knowing because it generalises: the t-statistic is self-normalising. It is a ratio, and the same outliers that inflate the sample mean in the numerator also inflate the sample standard deviation in the denominator. The two effects largely cancel. Self-normalised sums have far better limiting behaviour than their unnormalised counterparts — a known result, and a genuinely elegant one.

So: do not go into an interview claiming fat tails break your p-values. They mostly don't. What they break is something else.

What fat tails actually break

Stability. Take the real S&P returns from M3.1 and remove a handful of days:

   sample                        ann. return   ann. vol   Sharpe
   ──────────────────────────────────────────────────────────────
   all 2,513 days                    12.72%     18.13%     0.70
   drop 10 largest |moves|           13.09%     16.03%     0.82
   drop the 10 BEST days              6.21%     16.87%     0.37
   drop the 10 WORST days            19.54%     16.64%     1.17

Ten days out of 2,513 — four-tenths of one percent of the sample — move the Sharpe ratio between 0.37 and 1.17. That is the entire range from "not worth doing" to "excellent business," decided by which fortnight you happened to include.

The test statistic is well behaved; the quantity being estimated is hostage to a few observations. Which means an in-sample result that looks robust may simply reflect which extremes fell inside your window, and the out-of-sample period will contain different extremes. This is the honest fat-tail problem, and it is far more corrosive than a mis-sized p-value would be, because no amount of statistical rigour fixes it — only more data, which you cannot have.

You saw the same effect in M3.1: a lag-1 autocorrelation of −0.145 that collapses to −0.028 once 2020 is excluded.

Power, not size. The tests keep their false-positive rate, but their ability to detect a real effect degrades — you need more data than a Gaussian calculation suggests. Combined with M2.1's standard error of ~0.45 on a Sharpe estimate, the picture is consistent: not that you will see things that aren't there, but that you will be unable to confirm the things that are.

Significance was never the constraint

Now the part that matters more than any of it. Suppose your signal is statistically real. That tells you almost nothing about whether it is worth trading.

Let a signal have information coefficient IC — the correlation between its prediction and the next day's return. Over the same 2,513 days, with the S&P's real 114 bps daily volatility and a round-trip cost of 5 bps:

     IC    t-stat   gross bps/day   net bps/day   gross Sharpe
   ─────────────────────────────────────────────────────────────
   0.01      0.50         1.1           −3.9          0.16
   0.02      1.00         2.3           −2.7          0.32
   0.05      2.51         5.7           +0.7          0.79
   0.10      5.01        11.4           +6.4          1.59
   0.15      7.52        17.1          +12.1          2.38

Read the IC = 0.05 row carefully, because it is where most real signals live. It is statistically significant — t = 2.51, publishable, and it would survive a referee. It is also worth 0.7 basis points a day after costs, which is a rounding error that any slippage, any capacity constraint (M1.5), or any mild decay in the signal turns negative.

Meanwhile IC = 0.02 is not significant and loses money, and IC = 0.15 is a genuinely excellent signal that essentially nobody has in a liquid market.

The window in which a signal is both detectable and economically worthwhile is narrow, and its left edge is set by trading costs, not by statistics. Which gives the closing rule of M2 and M3 together:

Statistical significance is necessary and nowhere near sufficient. Ask for the effect size in basis points, then subtract the costs, before asking for the p-value.

The full checklist

Putting M2.7 and M3 together, when someone shows you a result:

   ✓ How big is the effect, in basis points, before any p-value?
   ✓ What are the costs at the intended size? (M1.5, M2.6)
   ✓ How many variants were tried before this one? (M2.7)
   ✓ How much of it rests on a handful of days? Drop the top ten
     and re-run — if it vanishes, you have an outlier, not an edge.
   ✓ Does it survive out of sample, on data looked at ONCE?
   ✓ Is the mechanism plausible, or is it only a correlation?
   ✓ What is the drawdown, not the variance? (M3.6)

The fourth line is M3's specific contribution, and it is the cheapest test on the list.

Where M3 leaves you

You can now say precisely why the standard toolkit misleads. Returns are not Gaussian — kurtosis 20, six-sigma days six times a decade. Tails follow a power law with α ≈ 3, so the higher moments you casually compute may not exist. Volatility clusters and is forecastable even though direction is not. Risk measures built on normality understate losses by 25% and get breached twice as often as advertised. Correlations rise exactly when diversification is needed. And position sizing, not signal quality, is what determines survival.

That is the module the roadmap said would reward a quantitative background most, and the reason is now clear: almost none of this is finance. It is heavy-tailed distributions, long-memory processes, extreme value theory, and multiplicative dynamics — statistical physics with a ticker symbol attached.

M4 changes register entirely, to macro and rates: less mathematics, more institutional knowledge, and the context that explains why the moves in this module happen when they do.

Source: Campbell Harvey’s presidential address, “The Scientific Outlook in Financial Economics” (2017), on significance thresholds in a field that tests thousands of hypotheses. For the self-normalisation result, Logan et al. (1973) is the original; any modern text on self-normalised sums covers it. Eric Falkenstein, The Missing Risk Premium, is the roadmap’s pick for the broader theme of results that are statistically real and economically absent.