Every Backtest Has a Start Date Somebody Chose

The same strategy over the same thousand days is a loser at an annualised Sharpe of −0.77 and a winner at +0.57. The only thing that differs between those two reports is where the window opens.

A backtest arrives as one number over one window. The number gets scrutinised. The window almost never does, because it rarely feels like a decision: the data began when the vendor file began, or in the year somebody finished cleaning it, or at the point where the series stopped containing an instrument that no longer trades.

Sometimes it is a decision, and a particular kind of one — somebody moved the start forward, looked at the curve, and moved it again. That is not sloppiness. It is a reasonable response to a sample that spans a market which has changed. But it is also a search, and a search has a count.

One strategy, twenty-five answers

A thousand daily returns. We evaluate the Sharpe ratio over the full sample, and then again after dropping the first month, the first two months, and so on out to the first twenty-four — each time keeping everything up to the present. Every figure below is annualised at 252 periods.

Read the first row and the seventh together. A researcher who reports the full sample is reporting a strategy that lost money. A researcher who starts the study twenty-three months later — for any of a dozen defensible reasons — is reporting a Sharpe of 0.57 on five hundred observations, which is not an obviously thin sample and not an obviously implausible figure.

Neither of them is lying. Only one of them will be published.

A window count is a trial count

The deflated Sharpe ratio exists to answer the question how many things did you try before this one. It is applied routinely to parameter sweeps and almost never to sample selection, because a parameter sweep produces a visible list of candidates and a moved start date produces one file.

The arithmetic does not care about that distinction:

There is a sharper way to put the same point. Given how much this metric already moves between windows, the expected maximum across twenty-five draws — on a strategy with no edge whatsoever — is 0.0541 per period, or roughly 0.86 annualised. The best window we actually found scores 0.0360, or 0.57.

What the tooling does

A sweep runs one metric across a set of windows and reports the spread, the sign changes and the trial count, in one command:

$ mdnorm windows pnl.csv --metric sharpe --trim-start 21 --count 24 --deflate
observations         1000
windows              24 (trimmed_start)
  shortest           496
  longest            979
full sample          -0.0485
lowest               -0.0509
median               -0.0325
highest              0.0360
spread               0.0869
positive             4/24 (16.7%)
note: the metric is positive on some windows and negative on others.
That is not a matter of degree.
trials               25
best window deflated
  as one trial       0.7934
  as 25 trials       0.3407

Four generators are provided and none is a default. Rolling windows ask whether a result holds in pieces. Expanding windows are what a walk-forward report looks like, and their last window is the whole sample, so that series is not independent of the headline. Trimmed starts ask how much of the result depends on the data beginning when it did. Trimmed ends ask the mirror question, and catch a result that depends on the last few weeks — including the weeks that arrived after somebody decided the strategy was working.

Three things we were careful not to claim

A wide spread is not proof of instability

A metric over half the data carries roughly √2 times the standard error, so part of any spread is sampling noise rather than a changing market. Separating the two needs a model of the return process, and this library does not have one. Every window is reported with its observation count so the shrinkage is visible, and the honest statement is that a wide spread is consistent with instability rather than evidence of it. That sentence is in the module documentation, not in a footnote.

The sensitivity is usually to a handful of observations

In the case above, the 483 observations that get dropped sum to −0.71 while the 517 that remain sum to +0.19. Nothing about a regime has been established. A bad two years was removed, and the phrase “the strategy works in the current regime” is doing a great deal of unearned work. Before concluding anything about regimes, look at what was in the part you dropped — a moved start date is often a discarded outlier wearing a different hat.

Trimming a sample is frequently correct

None of this argues for always using the longest available history. A sample that spans a market structure which no longer exists can be worse than a shorter one, and a researcher who trims for that reason is doing the right thing. The argument is narrower: the trim is a decision, decisions among alternatives have counts, and the count belongs in the deflation instead of in somebody's recollection.

Why there is no default

There is no default window length in this module, no default step and no default count. This is the same refusal that runs through the rest of the library: it will not guess how many strategies a search evaluated, it will not infer that a quiet stretch was a trading halt, and it will not decide how long a gap has to be before it counts as one.

A threshold chosen inside a library makes the answer partly a property of the library, and the person reading the result has no way of knowing which part. Requiring the caller to state it is less convenient by design. It also means the value ends up in a run manifest where a reviewer can find it.

Where to check any of this

Every figure above came from the windows module of our open-source library, released in version 1.37.0, run against the deflated Sharpe implementation in the same package. The series is a seeded pseudo-random draw — a thousand values from random.Random(23) at gauss(-0.00008, 0.011), rounded to ten places — so the whole table reproduces exactly on any machine, including the parts that are inconvenient for us. MIT licensed, no runtime dependencies, 1,473 tests, type-checked clean.

pip install market-data-normalizer

The source is on GitHub, the package on PyPI, and the reasoning behind what the library will and will not do is in ROADMAP.md beside the code. Longer write-ups are on our Medium, releases on LinkedIn, and the tooling we run against our own data is in the Console.

The rest of the series is in our blog: the file that did not change while the answer did, the stop-loss that always fills, the biggest trade of the day that was not a trade, a thousand rows and two hundred observations and the things we have decided not to build. Public comments on the work are quoted in full with a link to each source on the community page, and the team is on the about page.

We are open to everyone, from independent developers to funds. If something here is wrong, an issue with a concrete input and a statement of what the right answer would be is the most useful form to send it in — a failing test has always been worth more to us than a paragraph that is correct.

Frequently asked questions

Why is choosing a start date a trial?

Because it is a choice among alternatives that could have been reported instead. Twenty-four start dates a month apart are twenty-four candidate answers, and publishing the best of them is the same selection that publishing the best of twenty-four strategies performs. It is harder to see only because one strategy was ever written down.

Does this mean a long sample is always better?

No. A longer window can span a market that no longer exists, and trimming it is often the right call. The point is not that trimming is wrong, it is that trimming is a decision with a count attached, and the count belongs in the deflation rather than in somebody's memory.

How much of the spread is just noise?

Some of it, and nothing here separates the two. A metric over half the data carries roughly the square root of two times the standard error, so a shorter window moves more for reasons that have nothing to do with the market. Separating noise from instability needs a model of the return process, which this library does not have and does not pretend to. Every window is reported with its observation count so the shrinkage is visible.

What is the single most useful number in the report?

Whether the metric changes sign. A result that is profitable over some windows and loss-making over others differs from its neighbours by more than a matter of degree, and no averaging makes that disagreement go away.

Why does the library not choose a window length for me?

Because how long a window has to be before it says anything is a property of the series and of the metric, not of a library. A default would make the answer partly a property of the tool, and the person reading the result would have no way of knowing which part.

Is a wide spread proof that a strategy is unstable?

No, it is consistent with instability. That distinction matters, and we state it in the module documentation rather than in a footnote. What a wide spread does prove is that the headline figure was not determined by the strategy alone.

What should be done with the trial count once it exists?

Hand it to a deflated Sharpe ratio. In the case below the best window deflates to 0.79 treated as the only thing anybody looked at and to 0.34 once the twenty-five windows are counted, and the strategy loses money over the full sample either way.

Where can these figures be checked?

The windows module of our open-source library, released in version 1.37.0. The series is generated by a seeded pseudo-random draw stated in the article, so the table reproduces exactly on any machine. MIT licensed, no runtime dependencies, 1,473 tests.