market-data-normalizer 1.0: Composable Pipelines, NDJSON I/O and a Zero-Dependency CLI

Six weeks, ten tagged releases and 71 tests after the first commit, our open-source market-data library has reached its first stable version. Here is what shipped in 1.0, and why each piece exists.

We have written before about the hidden cost of dirty market data: duplicated ticks after reconnects, bad prints poisoning return series, missing bars silently shifting every feature window. market-data-normalizer (mdnorm) is our answer to that layer — a pure Python, zero-dependency library that turns heterogeneous feeds (CSV, exchange WebSocket JSON, FIX) into one exchange-agnostic schema: Decimal prices, nanosecond UTC timestamps, canonical symbols.

Version 1.0 is the release where the building blocks grow into a workflow.

1. Pipelines: Declare the Chain Once

Every ingestion job wires the same steps together: drop duplicates, clean bad ticks, aggregate to bars, resample, fill gaps. In 1.0 that chain becomes a declarative object you build once and run on any venue or file:

from decimal import Decimal
from mdnorm import Pipeline

pipe = (
    Pipeline()
    .dedupe()
    .clean(max_return=Decimal("0.1"))
    .time_bars(60_000_000_000)   # 1-minute bars
    .fill_gaps()
)
bars = pipe.run(events)
print(pipe.last_issues)          # data-quality report

The quality report is a first-class citizen: after every run, pipe.last_issues tells you exactly which ticks were dropped and why. A pipeline that cleans data silently is a pipeline you cannot trust.

2. NDJSON I/O: Speaking to Modern Data Stacks

CSV remains the lingua franca of quant research, but log shippers, object stores and streaming loaders speak newline-delimited JSON. 1.0 adds write_jsonl and read_jsonl_events — lossless round-trips in both directions, with Decimal precision preserved by default. No serialization drift between research files and production feeds.

3. A CLI for the Common Cases

Not every data task deserves a Python script. The most common conversions now run straight from the terminal:

$ mdnorm bars trades.csv --venue binance --interval 1m -o bars.csv
$ mdnorm quality trades.csv --max-gap 5m
$ mdnorm convert trades.csv -o trades.jsonl

mdnorm quality is the fastest way to answer a question every researcher should ask before training anything: can I actually trust this file?

What 1.0 Means Here

Every release from v0.1.0 to v1.0.0 is tagged in the repository with a documented changelog. The test suite runs on Python 3.10 through 3.13 in CI before anything ships, and the public API of the 0.x series carries over to 1.0 unchanged. Stability is a promise about process, not a version number — this is the process.

We open-source the primitives we rely on internally because data plumbing is a problem every quantitative team solves — and re-solves — in private. If your team is building on similar foundations, the code, the changelog and the discussion are all public: explore the repository on GitHub, read the companion deep-dive on LinkedIn, or reach out through our partnership page if you want to build with us.