Backtesting

Backtesting takes a rule stated precisely enough for a machine to follow, an entry condition, an exit condition and a size, and applies it to a stored series of past prices, recording every transaction the rule would have generated. The output is a set of statistics rather than a verdict: how many transactions the rule produced, what proportion closed in profit, the largest peak to trough fall in the resulting equity curve, and the net result after assumed costs.

The quality of a backtest is decided by its data and its assumptions, not by its result. Prices have to carry the spread and any commission that would actually have been paid, because a rule transacting often can look profitable on mid prices and unprofitable on the prices dealt. The series has to include instruments that were later delisted or merged, or the sample contains only survivors. And the rule may see nothing at each bar that had not been published by that bar, since a condition referring to a later close is arithmetic rather than a forecast.

The failure that catches most people is overfitting. Parameters tuned until the historical curve looks smooth describe that particular sample rather than the market, and the more settings a rule carries, the more certainly some combination of them will flatter the past. Practitioners disagree about the remedy. One school holds a portion of the history untouched and tests on it once; another argues that any history tested repeatedly is contaminated whatever is held back, and treats a backtest as a way of eliminating rules rather than of selecting them.