Problem 1.7 on pitfalls of Sharpe ratio

[adamny]

Problem 1.7 is very surprising: "You read a journal article that describes an investment strategy. In a backtest, it achieves an annualized Sharpe ratio in excess of 2, with a confidence level of 95%. Using their dataset, you are able to reproduce their result in an independent backtest. Why is this discovery likely to be false?"
Does anyone understand it? I understand that this could be the case if the strategy was a result of backtest overfitting, but there is no indication for it, is it?

[Despair2000]

I think MLDP is referring to the probably existing multiple testing bias.

[rspadim]

Number of backtests executed and considerations omitted should be exposed

[mturk24]

Experiment assumptions, number of simulations, optimal hyperparameter values (and results that aren't optimal) should be provided. Also, there may be an error in their sharpe ratio calculation (so it's worth checking their math/code)