Stats › Backtesting and evidence › Multiple testing

A new factor needs a t statistic above 3.0, not 1.96

Conventional single test
1.96
BHY, 5% false discovery rate
2.78
Their headline recommendation
3
Minimum once hidden tests are counted
3.18
Holm, 113 factors
3.29
BHY, 1% false discovery rate
3.39
Holm, 316 factors
3.64
Bonferroni, 316 factors, 2012
3.78
Bonferroni, forecast for 2032
4
Bonferroni once hidden tests are counted
4.01

Source: Harvey, Liu and Zhu, NBER working paper 20592. The p values they give alongside these hurdles are 0.27% for 3.0, 0.10% for 3.29, 0.07% for 3.39, 0.03% for 3.64, 0.02% for 3.78 and 0.01% for 4.00. The 2032 figure is their linear extrapolation of the factor count, not an observation. The 4.01 bar is Bonferroni after they model the tests that were run and never published.

A t statistic of 2 lets one test in twenty pass when nothing is there. Academic finance has published 316 factors, so the bar for a new one is a t above 3.0. Size, one of the best known factors, clears no adjusted hurdle.

316Factors published
1.96Conventional hurdle
above 3.0Recommended hurdle
3.78Bonferroni by 2012

Members only

The rest of this page is for members

Below this point there are 6 sections, 1 chart, 1 table and 4 named sources, roughly 1550 words of it. Every figure carries the source it came from and the date the data is from.

You can keep browsing every statistic in the library for free. The intro and the summary are always open.