Overfitting

A price record contains both whatever structure exists and a great deal of noise, and a rule with enough adjustable parts can fit both. As the number of things being adjusted approaches the number of observations available, the fit approaches exactness, and at that point the rule has become a compressed description of one particular history rather than a statement about markets.

It is not a mistake anyone has to be careless to make, which is the part most often missed. Repeated adjustment is the ordinary way a rule is developed, and each individual adjustment is defensible: a threshold moved because the previous value cut off an obvious case, a filter added because a known episode behaved differently. The record improves at every step, and nothing in the improving number distinguishes structure that will recur from noise that will not.

The conventional countermeasures are all forms of withholding evidence from the fitting process. Fewer adjustable parts, parameters that work across neighbouring values rather than at one setting, out of sample testing and forward testing on periods that had not happened when the rule was frozen. None of them proves a rule generalises. Each of them removes one route by which a rule can appear to.