What We Refuse to Build, and the One Refusal We Reversed

A roadmap tells you what a team intends to build. The list of things it has decided not to build tells you more, because each entry costs something and somebody had to write down why.

Our open-source market-data library has a section in its roadmap headed Decided against. It is short, it is public, and every item in it has been asked for at least once. This is that list with the reasoning spelled out, and then the one entry that did not survive.

1. A default annualisation factor

Multiplying a volatility by the square root of 252 is correct for daily bars on a market that trades 252 days a year. It is wrong for every minute bar, wrong for every instrument that trades around the clock, and wrong for the year in front of you, which held 251 sessions rather than 252.

The reason this one is worth refusing rather than defaulting is the shape of the error. A wrong constant here rescales every annualised figure in a report and leaves the shape of the result untouched. Nothing looks broken. The series has the same peaks and troughs in the same places; it is simply a slightly better or slightly worse strategy than it was.

2. A default tick size

A venue accepts multiples of a tick and nothing else. The reflex is to hard-code a penny, and it is wrong below a dollar on most venues, wrong for sub-penny programmes, wrong for crypto by several orders of magnitude and differently for every symbol on one exchange, and wrong for the same instrument before the last regime revision.

So a tick table is point-in-time data rather than a constant, and asking for one buys something back: prices that do not sit on the grid could not have been quoted, which is a one-pass test for whether a price file is real prints or derived numbers.

3. A backtest engine

This is the most requested thing we do not build. There are good engines already, and in our experience the reason a strategy fails in production is almost never the event loop — it is the data underneath, which is where the whole library lives.

The second reason is about what we would be asking of people. A library is something you call from the code you already have. A framework is something you restructure your code around. Those are very different commitments, and the second one is much harder to reverse if we turn out to be wrong about something.

4. Star ratings, leaderboards and comparisons with other libraries

We are not a neutral party about our own software. A benchmark we design, run and publish against somebody else's library is an argument wearing a measurement's clothes, and everyone reading it knows that.

The benchmark that does exist measures our own code against itself. It ships as a script in the repository, uses the standard library only, and reports the cases where we come off badly — including, in the run that prompted everything below, the discovery that the slowest thing in the library was our own algorithm rather than the exact arithmetic we had been blaming.

5. Bundling a data vendor

The library reads whatever you already have: a CSV dump, an exchange WebSocket feed, a FIX log. Tying it to one commercial feed would narrow it to whoever already pays for that feed, and the asymmetry between the people who can afford that and the people who cannot is the thing the project exists to reduce.

And the one we reversed

The benchmark file has contained this sentence since version 1.17.0:

The it was a sliding window sum. Recomputing a trailing total at every index costs O(n × window); sliding it — add the value arriving, subtract the value leaving — is O(n). It is the most obvious optimisation in the library and it was worth roughly an order of magnitude.

We refused because the library computes in exact decimals, and decimal addition rounds to the working precision. A running total therefore accumulates a different history of roundings than a fresh sum over the same values, and the two disagree in the last digits — with the disagreement depending on how far into the series you are. A library whose central claim is reproducibility cannot have a statistic that quietly depends on where the window sits.

That objection was correct. What it missed, for seven releases, is that the rounding is observable. The decimal type raises an inexact flag on precisely the operations that lose a digit.

flags[Inexact] = False
candidate = running + arriving - departing

if flags[Inexact]:
    running = sum(window)      # the update would have rounded
else:
    running = candidate        # provably the exact sum

The slid total is used only on the steps where it is provably exact, and every other step falls back to summing the window in full. The cost of knowing which case you are in is one flag read per point.

On ordinary data every output is unchanged. Where outputs do change, they change in one direction: summing a window forwards can round an intermediate partial that the slid total never holds, and there the slid total is the exact sum while the recomputed one lost a digit. The test suite checks that against exact rational arithmetic rather than asserting it. So the release is a correctness fix wearing a performance fix's clothes.

One thing we did not slide: the variance pass inside the standard deviation and the z-score. Sliding it means substituting an identity that is exact in algebra and a different sequence of roundings in arithmetic, which would change published numbers to save time. That is the trade we declined in 1.17.0 and still decline, and it is why the rolling mean moved 21× while the z-score moved about one per cent.

The longer version of this story, with the argument rather than the summary, is on our Medium.

What makes a refusal reversible

The useful part of this is not the optimisation. It is that we could check the refusal at all.

A decision recorded as a preference cannot be revisited, because there is nothing to test. A decision recorded with its reason can be, because the reason is a claim about the world, and claims about the world are the kind of thing that turns out to be measurable more often than people expect. Ours was written down as a running total rounds differently and you cannot tell when. The second half of that sentence was false, and it was only findable because somebody had bothered to write the sentence.

The paragraph refusing to do this is still in the benchmark file. We added the new section underneath it rather than editing it, because a record of the reasoning is worth more than a record of having always been right. The same file records an overstatement we published and corrected two days later.

Where to check any of this

Nothing on this page asks to be taken on trust, and that is deliberate. Every claim above has a place you can go and disagree with it.

The reasons live in ROADMAP.md and BENCHMARKS.md in the repository. The benchmark is a script in the same repository that runs on the standard library alone, so the numbers in the table above can be reproduced on your own hardware in a minute. The package is on PyPI, MIT licensed, with no runtime dependencies and 1,009 tests. The source is the part we would point at first.

The engineering write-ups that go with the modules are in our blog: reading a daily value before it existed, an index universe chosen using the end of the period, a trading year that was not 252 sessions and telling prints from arithmetic. Public comments about the work, each quoted verbatim with a link to where it was posted, are on our community page.

If you find something on this page that is wrong, the most useful form to send it in is an issue with a concrete input and a statement of what the right answer would be. A failing test is worth more to us than a paragraph that is correct — and, on the evidence above, more than our own settled opinions.

Frequently asked questions

Why does the library have no default annualisation factor?

Because no single factor is right for every market. The square root of 252 fits daily bars on a venue that trades 252 days a year and fits almost nothing else, including every minute bar and every instrument that trades around the clock. A wrong constant of this kind rescales every annualised figure in a report while leaving the shape of the result untouched, so it does not look like an error — it looks like a slightly better or slightly worse strategy. The calendar has to be stated, and the library computes the number from it.

Why is there no default tick size?

The familiar penny is wrong below a dollar on most venues, wrong for sub-penny programmes, wrong for crypto by orders of magnitude and different for every symbol on one exchange, and wrong for the same instrument before the last tick-regime revision. A tick table is point-in-time data, not a constant, so the library asks for one and refuses to answer for a period no table covers.

Why not ship a backtest engine?

There are good ones already, and the reason strategies fail is almost never the event loop — it is the data underneath. Adding an engine would turn this from a library you call into a framework you adopt, which is a much larger commitment to ask of someone who wants one function.

Why no star ratings or comparisons against other libraries?

We are not a neutral party about our own software. A benchmark we design, run and publish against a competitor is an argument wearing a measurement's clothes. The benchmark that does exist measures our own code against itself, ships as a script anyone can run, and reports the cases where we come off badly.

Why not bundle a data vendor?

The library reads whatever you already have. Tying it to one feed would narrow it to whoever already pays for that feed, which is the exact asymmetry the project exists to reduce.

What was the refusal you reversed?

A sliding window sum. Recomputing a trailing sum at every index costs O(n x window), and sliding it — add the arriving value, subtract the departing one — is O(n). We refused for seven releases because a running Decimal total accumulates a different history of roundings than a fresh sum, so the same window would stop giving the same answer. In v1.24.0 we did it anyway, because Decimal raises a flag on exactly the operations that round, which turns the objection into a check.

Did reversing it change any published numbers?

On ordinary price data, no — every output is unchanged value for value. There is a case where they differ: summing a window forwards can round an intermediate partial that the slid total never holds, and there the slid total is the exact sum while the recomputed one lost a digit. The test suite verifies that against exact rational arithmetic rather than asserting it.

How can I check any of this myself?

Everything named here is public. The reasons live in ROADMAP.md and BENCHMARKS.md in the repository, the benchmark is a script in the same repository that runs on the standard library alone, the package is on PyPI, and the test suite is the part we would point at first. Nothing on this page asks to be taken on trust.