The limits of testing on history

What the procedure actually produces 

A test on history is arithmetic applied to a stored record. It takes a series of prices, a set of rules restated as conditions on that series, and a set of assumptions about what would have happened when a condition was met, and it produces a ledger of positions that never existed. The output is a description of one sample of the past under one set of assumptions. It is not a measurement of the rule, because a rule has no fixed property waiting to be measured. It is a description of what the rule, the sample and the assumptions produced together.

That is why a favourable description is so difficult to interpret. Three separate things can produce one: a rule that captured something durable, a sample that happened to suit it, or a set of assumptions generous enough to flatter anything. The result is a single figure, and nothing in it says which of the three it came from. The sections below take each in turn, because the failures are not exotic and none of them requires anyone to be careless.

Repeated adjustment fits the noise 

Overfitting arises from the search, not from bad faith. Every adjustment described in the opening paragraph was made after seeing a result, which means each one was chosen because of how the sample responded to it. A parameter selected because it scored highest on a record has been fitted to that record, including the parts of it that carry no structure at all: the particular week a gap fell in, the particular order in which two ordinary days arrived, the one instrument whose worst month happened to land inside the test window.

The scale of the problem is combinatorial, and it grows faster than it looks. Each parameter that is varied multiplies the number of candidate rules the search covers, and the one that is kept is the one that scored highest across all of them.

That last point is the one worth holding on to. The highest score in a large search is partly a measurement of the size of the search. The more combinations examined, the higher the best of them is expected to score on the sample examined, and the less that score has to do with the rule. The same arithmetic works against a rule with many conditions tested on a record containing few positions: as the number of things being adjusted approaches the number of observations available, a rule can be shaped to describe the record almost exactly, and a description of a record is not a rule.

Historical fills are assumptions, not records 

A price history contains prices. It does not contain deals. When a test reports that a position opened at a level, no deal took place at that level, and none was refused there either. Somebody decided that a deal would have happened, at that price, in that size, at that instant. Every one of those is an assumption, and the assumptions are ordinarily written by whoever is running the test, which is ordinarily somebody who would prefer the result to be favourable.

The cleanest illustration is a single bar that contains both the stop level and the exit level. A stored bar is an open, a high, a low and a close, and the order in which the high and the low were reached is not part of the record. The test has to assume one, and whichever it assumes is then applied to every bar of that shape across the whole history.

The same class of assumption sits under everything else a test reports about execution:

• The price a stop is filled at. A test fills it at the level. A market fills it at the next available price, which in a gap or a fast market is beyond the level, as an earlier lesson in this module set out.
• Resting orders that were never queued. A test treats a level as filled once price touches it. In a market, a touch means somebody traded there, not that every order sitting at that price was reached.
• Which side of the quote each end used. Stored histories are commonly a single series, a mid or a bid. A position opens on one side of the quote and closes on the other, so a test run on one series has priced both ends on the wrong side unless the spread was added deliberately.
• The interval between the condition and the deal. A rule that acts on a bar's close assumes a deal at the same price that defined the condition, in the same instant the condition became knowable.
• Size. A test can place any size at a printed price. A printed price stands for a limited quantity, and the rest of an order fills at whatever is behind it.
• The moments the record is thin. Session breaks, weekends, holidays, halted instruments and periods when a market was quoted but barely traded all appear in a history as ordinary prices.

The frictions a test leaves out 

Cost is the most commonly omitted input and the most predictable in its effect, because it is not a random error. It runs one way. A test that ignores the spread crossed at both ends, the commission charged at both ends and the financing adjustment on anything held past a daily cut off overstates every single position in the ledger by roughly the same amount, and the overstatement accumulates with the number of positions rather than cancelling out across them.

Two consequences follow. The first is that the omission scales with turnover, so the shorter the average holding period, the larger the omitted friction is relative to the distance the position was supposed to travel. A rule tested without costs and the same rule tested with them are not variants of one another; past a certain rate of turnover they describe different things. The second is that a typical cost is not the cost that applied at the moment the rule traded. Spreads widen around scheduled releases, at session opens and into a close, which are exactly the moments many rules are written to act on, so applying an average charge to those positions understates them specifically.

The record the test was run on 

A test needs a list of instruments, and the obvious list is the one that exists today. Everything that stopped existing is missing from it: shares that were delisted or taken over, funds that closed, contracts that were discontinued, currency arrangements that were abandoned. A test over today's list has run over the survivors of the period it covers, and it inherited that selection silently rather than choosing it.

The same effect reaches the composition of an index, whose membership is revised as companies enter and leave, so a test run over today's members has used knowledge that was unavailable throughout most of the record. It reaches the prices too: share histories adjusted for corporate actions differ from the prices that were actually quoted, and a series stitched together across contract rolls contains a step at each roll that no position ever experienced. None of this appears as a warning in the record. It appears as ordinary numbers.

There is a subtler version that no dataset can correct. Anyone testing a rule has lived through part of the period being tested and has read descriptions of the rest, and the rules considered worth testing at all are largely those that survived being written about. A condition that excludes a particular month was written by somebody who knows what happened in that month. That knowledge is not in the data, so a test cannot subtract it, and it flatters the result exactly as an explicit adjustment would.

Holding data back, and what it settles 

The conventional response to all of this is to partition the record: fit on one portion, then examine the result on a portion that was never looked at while the rule was being shaped. Rolling that partition forward through the record, refitting on each earlier window and examining the next, is the more demanding version of the same idea.

The limit is structural, and practitioners who use the technique are usually the first to state it. A held-back portion is genuinely held back once. The moment a rule is adjusted after seeing how it fared there, that portion has joined the fitting, and it cannot be restored to its earlier status by intending not to look again. In ordinary practice the partition is consulted repeatedly, and after enough consultations the distinction between the two portions is nominal rather than real.

A second limit is that both portions come from the same record, and a record is a period. An interest rate environment, a currency arrangement or a volatility regime that persisted from one end of the history to the other is a constant inside the test rather than a variable, so no partition of that history can examine what the rule does when it changes. A test is silent on every condition its record does not contain.

Where practitioners disagree 

Two positions are held seriously, and the disagreement is genuine rather than a matter of rigour. One holds that testing is indispensable, because a rule never run over anything has unknown behaviour, and that the discipline lies in constraining the search: few parameters, values chosen for reasons stated before the test rather than after it, and a preference for a broad plateau of similar results over an isolated peak, on the grounds that a peak is what a fitted parameter looks like. The other holds that testing on history mostly manufactures confidence out of proportion to what was learned, and that its one defensible use is disqualification.

Both sides agree on the asymmetry underneath that second claim, and it is the most useful thing in this lesson. A poor result is far more informative than a good one. There are few ways for a rule to fail by accident across a long record, and a great many ways for one to succeed by search. Testing can therefore rule things out with reasonable confidence and rule almost nothing in. Practitioners also disagree about how much history is enough, since what matters is the number of distinct conditions the record contains rather than the number of days in it, and about whether running a rule forward in a simulated account settles anything, since that procedure takes exactly as long as the market takes.

Why no historical result is published here 

The Academy publishes no historical result for any rule, pattern, indicator or method, and the reason is the content of this lesson rather than a formality. A result of that kind is a description of one sample, produced under assumptions chosen by whoever ran it, by a rule selected from a search whose size is not disclosed. A reader cannot inspect the sample, the assumptions or the search, and without all three the figure carries no information. Presented anyway, it stops being a description of the past and starts functioning as a claim about the future, which nobody is in a position to make.

The same reasoning applies to a result produced privately, and it is the harder case, because a figure someone generated themselves feels earned in a way that a published one does not. The arithmetic does not know who ran it. A test describes what a set of rules and a set of assumptions did to a stored record, and the honest way to read one is as a source of questions about a rule rather than an answer about it.