Out of sample testing

A partition of the record into a stretch used for development and a stretch reserved for assessment. The rule is shaped, adjusted and finalised on the first stretch only; the second is opened once and the frozen rule is run over it. The more demanding version rolls the partition forward through the record, refitting on each earlier window and examining the next, which produces a sequence of assessments rather than one.

It exists to answer a question a plain backtest cannot. An improvement produced by adjustment carries no information about which of two things produced it, the rule or the record, and once the number of things being adjusted approaches the number of observations available, a rule can be shaped to describe the record almost exactly. A description of a record is not a rule, and the held back stretch is the only evidence available that the description generalises.

The limit is structural, and the practitioners who use the technique are usually the first to state it. A held back stretch can only be used once: the moment a disappointing result sends anyone back to adjust the rule, that stretch has entered the development sample and the partition no longer means what it did. Nothing in the method prevents that, which is why the discipline is a matter of record keeping rather than of arithmetic.