The File Did Not Change. The Answer Did.

Two runs of one pipeline over one file produced a result worth publishing and a result worth discarding. The data was byte-identical in both. Nothing anywhere recorded the number that separated them.

This library has spent a lot of releases refusing to guess. It will not invent the number of strategies a search evaluated, it will not pick a threshold for how long a silence has to be before it counts, it will not infer that a quiet stretch was a trading halt, and it will not choose a truncation lag for you. Every one of those refusals hands a decision to the person running it.

Last week it occurred to us that nothing, anywhere, was writing down what they decided.

One file, two answers

Here is the case in full. A thousand daily returns, an annualised Sharpe ratio of 0.992, skewness of +0.027 and kurtosis of 2.708. A perfectly ordinary result, and on its own a mildly encouraging one.

It only means something once you know how many strategies were tried before this one was chosen, so the deflated Sharpe ratio takes that count as an input:

That is the whole problem in one table. The strategy did not change, the data did not change, the code did not change. An argument changed, and the argument was the difference between a number somebody would put in a deck and a number somebody would delete.

What a checksum cannot see

The standard answer to reproducibility is to hash the inputs. It is a good answer and it is not sufficient, and this case shows exactly where it stops:

the run       fingerprint c3120b979e7a   input digest ddaa2d1032e5\nthe write-up  fingerprint 683c332b703e   input digest ddaa2d1032e5\n\nsame file  : True\nsame run   : False

One file, hashed twice, identical both times — because it is identical. A checksum-only check reports that pair as reproducible, and it is telling the truth about the data and something false about the result.

An input that changes is usually noticed, because a person had to change it. Somebody re-exported the file, or the vendor restated a row, and there is a trace. An argument that changes leaves no trace at all: it was typed on a command line in March and typed differently in April, and no file on disk is any different for it.

What we built

A manifest records what a run read, what it was told and what produced it: a content digest per input, the arguments beside them, the library version and the Python version. A few hundred bytes of JSON written next to the output.

$ mdnorm provenance run.json --verify --parameter trials=50\ncommand              sharpe\nrecorded with        market-data-normalizer 1.35.0\nfingerprint          c3120b979e7a\ninputs               1\nresult               1 difference(s)\n  pnl.csv: parameter '500' -> '50'\nnote: nothing here says which side is right. A changed input may be a\ncorrection or a corruption, and that is not a question this library\ncan answer.

The exit status is non-zero when a run does not reproduce, so this goes into a scheduled job without anybody writing a parser for it.

Four decisions worth defending

The clock is not part of the fingerprint

Two runs of the same pipeline over the same inputs with the same arguments must fingerprint identically, or the fingerprint answers no question worth asking. Including a timestamp makes every run unique, which is exactly the property that would render it useless. The free-text note is excluded for the same reason: a comment added after a meeting does not change any number.

Neither are the outputs

This one looks wrong until you state the question. A fingerprint answers was this the same run, and a run is defined by what went into it. Two identical fingerprints with different results is not a contradiction — it is the finding, and covering the outputs would hide it by construction.

Float parameters raise

A parameter of 0.1 is not a value, it is a rendering of one, and the rendering differs by platform and by Python version. A manifest that accepted it would be promising to reproduce something it can only approximate. Strings, integers, booleans and decimals are accepted; a float is refused, with a message that says why rather than a type error.

An edited manifest is refused

Reading one re-derives the fingerprint from the contents and compares it against the value written inside. A mismatch raises. A manifest somebody has quietly corrected is worse than no manifest at all, because it carries the authority of a record while stating something that never happened.

What it deliberately will not do

It does not judge. The verification reports what moved and stops there. Whether a changed input is a vendor correction or a corrupted download is not a question a library can answer, and a tool that decided would start being trusted for a judgment it is not entitled to make.

It also does not count rows, which surprises people. Counting rows means parsing, parsing means a schema, and a manifest that had to understand its inputs could not record an input it did not understand. The row count is accepted if you supply it and never computed.

A name collision, and why it is in the changelog

The module first shipped internally with a type called ChangeKind. So does the index-membership module, which has exported one under that name since version 1.19.0 for additions and deletions. Importing both into the package namespace silently shadowed the older one: nothing failed, no test broke, and a type that had been reachable for sixteen releases simply stopped being reachable.

It was caught before release, renamed, and the release notes say so under Fixed rather than not mentioning it. There are now three package-level tests that no exported name is duplicated, that every name resolves, and that the two types stay distinct — which is the only part of this story that will still matter in a year.

We write these down because a changelog that contains no mistakes is either a very short changelog or an edited one.

Where to check any of this

Every figure above came from the provenance module of our open-source library, released in version 1.35.0, run against the deflated Sharpe implementation in the same package. MIT licensed, no runtime dependencies, 1,400 tests, type-checked clean.

pip install market-data-normalizer

The source is on GitHub, the package on PyPI, and the reasoning behind what the library will and will not do is in ROADMAP.md beside the code. Longer write-ups are on our Medium, releases on LinkedIn, and the tooling we run against our own data is in the Console.

The rest of the series is in our blog: the stop-loss that always fills, the biggest trade of the day that was not a trade, a thousand rows and two hundred observations, what AI changed and what it multiplied and the things we have decided not to build. Public comments on the work are quoted in full with a link to each source on the community page, and the team is on the about page.

We are open to everyone, from independent developers to funds. If something here is wrong, an issue with a concrete input and a statement of what the right answer would be is the most useful form to send it in — a failing test has always been worth more to us than a paragraph that is correct.

Frequently asked questions

Why is hashing the input files not enough?

Because the inputs are rarely what moved. In the worked example the file is byte-identical across both runs — digest ddaa2d1032e5 in each — and the published figure goes from 1.0000 to 0.0046 because a trial count changed from 500 to 50. A checksum of the data reports that run as reproducible, which is true of the data and false of the result.

What is a run manifest?

A record of what a run read, what it was told and what produced it: a content digest per input file, the arguments beside them, the library version and the Python version. It is a few hundred bytes of JSON written next to the output, and it is the difference between a result and an anecdote.

Why exclude the timestamp from the fingerprint?

Because two runs of the same pipeline over the same inputs with the same arguments have to fingerprint identically, or the fingerprint answers no question worth asking. A clock makes every run unique, which is precisely the property that would make it useless.

Why are the outputs excluded as well?

The question a fingerprint answers is whether this was the same run, and a run is defined by what went into it. Two identical fingerprints with different results is not a contradiction — it is the finding, and including the outputs would hide it by definition.

Why does a float parameter raise instead of being recorded?

Because 0.1 is not a value, it is a rendering of one, and the rendering differs between platforms and Python versions. A manifest that accepted it would be promising to reproduce something it can only approximate. Strings, integers, booleans and Decimals are accepted; a float is refused with a message saying why.

What happens if somebody edits the manifest?

Reading it fails. The fingerprint is re-derived from the file's contents and compared against the one written inside it, and a mismatch raises. A manifest corrected by hand is worse than no manifest at all, because it carries the authority of a record while stating something that never happened.

Does the tool decide whether a change was legitimate?

No, and that is deliberate. It reports what moved and stops. Whether a changed input is a vendor correction or a corrupted download is not a question a library can answer, and a tool that decided would be trusted for a judgment it is not entitled to make.

How does this fit into a review or due-diligence process?

The manifest is what turns 'we ran it in March' into a checkable statement. The command exits non-zero when a run does not reproduce, so it goes into a scheduled job without any parsing, and the reviewer's question changes from whether the pipeline was stable to which specific input or argument moved and when.