What a p-value is
The probability of observing data at least as extreme as yours, assuming the null hypothesis is true.
What it is not
It is not the probability that the null hypothesis is true. That conflation is the most common error in applied statistics, and interviewers ask about it directly.
Getting from one to the other requires a prior and Bayes' theorem. If you test a hypothesis that is a priori unlikely, a p-value of 0.05 leaves it still probably false - exactly the structure of the rare-disease problem.
Multiple testing
Test 20 independent strategies at the 5% level with no real edge in any of them, and you expect one to look significant. Test 1,000 and you expect 50.
This is not a subtlety - it is the central problem in quantitative research, and interviewers care much more about whether you understand it than about test mechanics.
Backtesting is multiple testing. Every parameter you tune, every variation you try, every date range you check is another test. A strategy that looks good after 200 attempts is exactly what you would expect from noise.
Corrections: Bonferroni (divide the threshold by the number of tests - conservative), false discovery rate (less conservative, usually more appropriate), and most importantly out-of-sample validation on data you have not touched.
The deeper problem
Even a held-out set stops being out-of-sample once you have used it to make decisions several times. The discipline of genuinely reserving data - and being honest about how many things you tried - is what separates real research from data mining.
Statistical versus economic significance
With enough data, trivially small effects become statistically significant. A signal with a p-value of 0.001 and an expected return below transaction costs is worthless.
Always ask both questions: is it real, and is it big enough to matter after costs.
The interview form
"You backtested 100 strategies and the best has a t-statistic of 3. Is it real?" The expected maximum t-statistic from 100 pure-noise strategies is around 2.5 to 3, so a 3 is unremarkable. The right answer names the multiple-testing problem and asks how many were tried.
Practise in statistics.