Market Data Normalization: A Practical Guide for Quantitative Teams

Every quantitative result rests on a layer nobody writes papers about: the pipeline that turns a raw exchange feed into a dataset you can trust. This guide walks that path end to end — schema, deduplication, quality, sampling, sessions and storage — and explains why each decision matters.

Market data normalization is the process of converting feeds from different venues and formats into one consistent schema, so downstream code never has to know where a record came from. It sounds like plumbing because it is plumbing — and it is also where a large share of quantitative errors originate. A model trained on inconsistent data learns the inconsistencies.

Four ways a raw feed breaks research

Before designing a schema, it helps to know precisely what goes wrong. Four failure modes account for most of the damage, and we covered their symptoms in detail in The Hidden Cost of Dirty Market Data.

Format divergence. The same trade arrives as a CSV row with an ISO-8601 timestamp, a WebSocket message with millisecond epochs and single-letter keys, and a FIX execution report with numeric tags. Each has its own idea of what a symbol looks like.

Numeric drift. Prices stored as binary floating point accumulate representation error. One print is fine; ten million aggregations later, VWAP no longer reconciles with notional turnover.

Temporal disorder. Feeds arrive out of order, exchange clocks disagree, and every reconnect replays messages you already stored. Merged naively, the consolidated tape double-counts fills and occasionally travels backwards in time.

Silent absence. A dropped connection produces missing minutes that look identical to genuinely quiet minutes. Every rolling window computed over that series is quietly misaligned.

The canonical schema: three decisions that matter

A normalized event needs only a handful of fields, but three choices determine whether the rest of the stack is sound.

Decimal, not float, for money. Fixed-point arithmetic eliminates an entire class of reconciliation bug. The overhead is irrelevant next to network and storage costs.

Integer nanoseconds in UTC. A single integer per timestamp, with no timezone ambiguity and no floating-point seconds. Local time is a presentation concern, applied at the edges — never stored.

One symbol convention. Traded pairs normalize to a BASE-QUOTE form, so BTCUSDT, XBT/USD and btc_usd resolve to one identifier. Single-listed instruments — equities, ETFs, indices — have no quote leg and keep their ticker. Skip this step and cross-venue joins return partial results without raising a single error.

Deduplication and cross-venue consolidation

Once events share a schema, multiple venue feeds can be merged into one timestamp-ordered timeline. Two rules keep that merge honest. Deduplication must compare the full event identity — venue, symbol, timestamp, price, size and side — because at nanosecond resolution legitimate distinct trades routinely share a timestamp, and dropping by timestamp alone destroys real data. And the merge must be stable: when two venues report the same nanosecond, their relative order should be deterministic across runs, or the same backtest will produce different numbers on Tuesday than it did on Monday.

Quality checks that run before the model

A pipeline that cleans data silently is a pipeline you cannot trust. Cleaning should always return a report: which records were dropped, at which positions, and for what reason. Four checks catch most real-world damage — tick-to-tick returns beyond a plausible threshold, timestamps that move backwards, forward gaps longer than a configured tolerance, and non-positive prices or sizes.

The distinction that matters is between removal and annotation. Outliers and impossible values can be dropped. Gaps and ordering problems should be reported but left in place, because only the researcher knows whether a five-minute silence at 03:00 is a data incident or an accurate description of an illiquid market. This is also the layer where most of the leakage discussed in The Overfitting Trap is prevented — long before a model is fit.

Choosing a sampling clock

Aggregating ticks into bars is a modelling decision disguised as a formatting step. Time bars sample on a wall-clock schedule, which is convenient, familiar and statistically awkward: a dead hour and the minute after a CPI release carry equal weight, so volatility clusters and information arrives in clumps.

The alternatives sample by activity instead. Tick bars close after a fixed number of trades, volume bars after a fixed traded quantity, and dollar bars after a fixed traded value. All three produce returns closer to the independent, identically distributed world that statistical models assume, and dollar bars additionally stay comparable as prices change over long samples.

from decimal import Decimal
from mdnorm import Pipeline, US_EQUITY_RTH

pipe = (
    Pipeline()
    .dedupe()
    .clean(max_return=Decimal("0.1"))
    .session(US_EQUITY_RTH)      # regular hours, New York
    .dollar_bars(Decimal("1e6")) # one bar per $1M traded
)
bars = pipe.run(events)
print(pipe.last_issues)          # what cleaning removed, and why

Trading sessions, time zones and daylight saving

Markets do not trade around the clock, and research that ignores session boundaries mixes regimes that behave nothing alike. Pre-market crossings, auction prints and overnight thin trading all distort features intended to describe regular hours.

Session filtering has two traps. The first is daylight saving: a 09:30 New York open is 13:30 UTC in summer and 14:30 UTC in winter, so any filter written as a fixed UTC offset is wrong for several months a year. The correct approach is to express the window in exchange-local time and let a timezone database resolve it. The second is overnight sessions — futures venues commonly open at 18:00 and close at 17:00 the following day — where the entire night must be attributed to the trading day on which it opened, or a single session is split across two buckets.

$ mdnorm bars trades.csv --interval 5m \
      --session 09:30-16:00 --tz America/New_York -o rth.csv

Storage and interchange

The last mile is unglamorous and consequential. CSV remains the lingua franca of quantitative research and survives every tool. Newline-delimited JSON suits log shippers, object stores and streaming loaders, and preserves structure better. Whichever you choose, two properties matter: round-trips must be lossless — a decimal written and read back must be the same decimal, not an approximation — and compression should be transparent, because tick data is measured in gigabytes and nobody should write a separate code path for compressed inputs.

Where normalization sits in the wider stack

Normalization is one layer among several. Above it sit the research and execution systems; below it, the capture path we described in What HFT Infrastructure Looks Like — colocation, feed handlers, kernel bypass and hardware timestamping. The latency characteristics of that capture path determine what your data can possibly represent, and the language boundary between C++ and Python usually falls exactly at the normalization line: hot-path capture in a systems language, normalization and research in Python.

We build this layer in the open. market-data-normalizer implements everything described above — unified schema, CSV, WebSocket and FIX normalizers, quality reporting, four bar types, session filtering, and CSV/NDJSON I/O with transparent compression. It is pure Python with no runtime dependencies, MIT-licensed, and every release is tagged and tested in public: github.com/Harvestgroup360/market-data-normalizer. The 1.0 release is described in this write-up.

If your team is building on the same foundations, our data infrastructure covers the layers above this one, and the partnership page is the way to start a conversation.

Frequently asked questions

What is market data normalization?

Market data normalization is the process of converting feeds from different venues and formats — CSV exports, exchange WebSocket messages, FIX execution reports — into one consistent schema, so that downstream research and execution code does not need to know where a record came from. In practice it covers four things: a single event structure, one numeric representation for prices and sizes, one timestamp convention, and one symbol convention.

Why should market data prices use Decimal instead of float?

Binary floating point cannot represent most decimal fractions exactly, so prices and sizes accumulate small errors under arithmetic. Those errors are invisible in a single print and material after millions of aggregations: VWAP drifts, notional sums stop reconciling, and comparisons that should be exact fail. Fixed-point decimal arithmetic removes that class of error entirely, and the cost is negligible relative to the network and storage layers around it.

What is the difference between time bars and dollar bars?

Time bars sample the market on a wall-clock schedule — one bar per minute, hour or day — so a quiet overnight period and a post-announcement burst receive the same weight. Dollar bars close whenever a fixed amount of traded value has accumulated, so bars form quickly during active periods and slowly during quiet ones. Returns sampled by activity tend to be closer to the independent, identically distributed assumption most statistical models rely on, which is why tick, volume and dollar bars are standard tools in quantitative research.

How do you handle gaps in a bar series?

First decide whether the gap is real. A missing interval can mean no trading occurred, or that the feed disconnected. If the series must be continuous — most feature pipelines assume it is — the standard treatment is to insert a synthetic bar whose open, high, low and close all equal the previous close, with zero volume and no VWAP. That keeps every rolling window aligned to real time while making the absence of trading explicit rather than silent.

What causes duplicate ticks in market data and how do you remove them?

Duplicates come from reconnects and replays: after a dropped connection, most venues resend a short window of recent messages, and naive consumers append them twice. They also appear when two capture processes write to the same store. Removing them requires comparing the full event identity — venue, symbol, timestamp, price, size and side — rather than the timestamp alone, because legitimate trades frequently share a timestamp at nanosecond resolution.

How do you restrict market data to regular trading hours?

Filter events by a session window expressed in the exchange's local time, not in UTC. A 09:30 to 16:00 New York session corresponds to 13:30-20:00 UTC in summer and 14:30-21:00 UTC in winter, so any filter written in fixed UTC offsets breaks twice a year. Sessions that cross midnight — an 18:00 open closing at 17:00 the next day — additionally require that the whole night be attributed to the trading day on which it opened.

What is a canonical symbol format?

A canonical symbol format is a single spelling that every venue's naming is mapped onto. Traded pairs are usually normalized to a BASE-QUOTE form, so BTCUSDT, XBT/USD and btc_usd all resolve to one identifier. Single-listed instruments such as equities, ETFs and indices have no quote leg and keep their plain ticker. Without this step, joining data across venues silently produces partial results.

Does HarvestGroup360 provide open-source market data tools?

Yes. HarvestGroup360 develops market-data-normalizer, an open-source Python library that implements the normalization pipeline described in this guide: a unified event schema, trade and quote normalizers for CSV, WebSocket and FIX inputs, data-quality reporting, time, tick, volume and dollar bars, session filtering, and CSV/NDJSON input and output. It is MIT-licensed, has no runtime dependencies beyond the Python standard library, and every release is tagged and tested in public at github.com/Harvestgroup360/market-data-normalizer.