Every Name on Today's List Is a Name That Survived

The index composition you can download for free is the one that exists now. Run a study on it and the sample was chosen using information from the end of the period — twice over, in opposite directions.

Getting the current members of a major index takes about ten seconds. Getting the members as they stood four years ago is a different exercise, and for most people it does not happen — so the current list gets used for the whole history, and the study inherits a sample selected for having survived.

This is a well-known problem with a badly-known size. It is also, in the way it is usually described, only half the story.

Both directions, and they add up

The familiar half is the removals. A company that left the index is absent from today's list, so it never enters the study, and companies rarely leave because things were going well. The sample loses its worst outcomes.

The half that gets left out is the additions. Today's list also contains the names that joined recently, and running it backwards holds them through years when they were not members — years during which they were, typically, doing the growing that earned them the place. The sample gains its best outcomes early.

How large, on a worked example

The size depends entirely on turnover and distance, so it is worth computing rather than carrying around as a feeling. Here is a synthetic index, deliberately simple so the arithmetic can be checked by hand: a hundred members, five years, five replacements a year, each replacement announced a week before it takes effect.

Nothing about it is a market. It is a shape — a hundred names with four percent annual turnover — and the point is what that shape costs when the universe comes from the end of the period rather than the middle of it.

Forty wrong names out of a hundred is not a correction to apply at the end. It is a different universe. And the figure grows with exactly the thing that makes a backtest look serious — a longer history.

The two dates, and the one that gets picked by accident

Underneath the survivorship question is a smaller one that is easier to get wrong. Every index change carries two dates: the day it was announced, and the day it took effect. In the example above they sit a week apart, which is typical.

Both are real. They answer different questions. Who was in the index on a given Wednesday is the effective date. When the change became public — when a strategy could first have acted on it — is the announcement. Three days before a change takes effect, the two disagree about both names involved:

A study that ranks its universe on one basis and trades the effect of the other has used a timestamp to justify a decision the other timestamp governs. It passes every check, because both columns are correct.

What a periodic list cannot tell you

Most vendors ship member lists rather than a record of changes — here are the constituents as of this date, monthly or quarterly. A list can show that somebody is gone. It cannot show when they went.

Reconstructing a history from monthly snapshots of the index above recovers all fifty changes, and dates each of them only to within thirty days. That is longer than plenty of holding periods, and it is the difference between a record and an estimate.

Two things follow. Date each inferred change at the later snapshot rather than the earlier one, because that never claims membership before the data supports it. And carry the width of the window with the change instead of discarding it, so whoever reads the result knows which parts were recorded and which were deduced.

Two smaller traps in the same file

Membership belongs to instruments, not tickers. A ticker is a label the exchange reassigns, and a history keyed by ticker stacks an identity error on top of a survivorship one. That problem has its own shape and its own fix, covered in building a point-in-time symbol map.

Re-entry is two spells, not one. Names leave an index and come back. Flattening that into a single membership marks an instrument as a member through a gap it spent outside — and the gap is frequently the interesting part, because whatever put it out and brought it back is exactly the kind of event a strategy is looking for.

Validating a file you were handed

Before building anything on an index file, three questions settle what it can support.

Did anything ever leave? If the answer is nothing, it is a list of current members and cannot answer a point-in-time question at all. This is the single fastest check and it disqualifies more files than people expect.

Do changes carry two dates? If only one is present, then the announced and effective bases are identical in that file, and any study of the announcement effect built on it is measuring the effective date under another name.

How much of it was inferred? If the history came from snapshots, the answer is every change in it, each dated to within the gap between files.

Version 1.19.0 of our open-source library answers all three from the command line, and says the first one out loud:

$ mdnorm membership index_changes.csv --at 1770000000000000000

instruments          125
changes              150
  additions          125
  deletions          25
no announcement      0
announcement cover   100%
members at ... on the effective basis: 100
a today-list would drop  20
a today-list would add   20

On a file with no departures it prints a note rather than a number: nothing ever left this index, so it is very likely a list of today's members rather than a record of who was in it when. The library keeps the two bases apart and makes the caller name which one governs, refuses to pick a date inside a window a snapshot only bounds, and hands the result to the universe machinery so a cross-section can be masked to the index as it stood.

Where this sits

This is the same failure the last few of these write-ups keep arriving at, in a new place. A join that reads a bar before it closed, a daily value read at its label, a figure revised after publication, and now a universe assembled from the end of the period. Every one of them produces numbers that are real, dated correctly, and unavailable at the time they were used. None of them fails loudly, and all of them improve the result.

The library is MIT licensed, has no runtime dependencies, and the membership module closes a roadmap item that had been open since we first published the file. What we have decided against is in the same document. Comments people have left on the work, each linked to its original, are on our community page.

If you run the check on your own index file and the answer to the first question is nothing ever left, we would like to hear which vendor it came from. That is more useful than any argument about methodology.

Frequently asked questions

What is survivorship bias in a backtest?

It is what happens when the universe you test on was chosen using information from the end of the period. Take today's index members and run them back five years and every name in the sample is a name that lasted five years. The companies that were removed are missing, and companies are rarely removed while things are going well. Nothing in the data looks wrong, because every price and date in it is real.

Does survivorship bias only remove the losers?

No, and this is the part that is usually left out. A today-list also contains the names that joined the index recently, and running it backwards holds them from before they joined — through the period when they were growing into the place they eventually earned. One direction removes losers and the other inserts winners early, so the two add up instead of cancelling.

How large is the effect?

It depends entirely on turnover and how far back you go, which is why it is worth computing rather than assuming. In the worked example on this page — a hundred-name index with five replacements a year — a today-list is wrong about forty of the hundred names four years back, twenty in each direction. At one year back the same index is wrong about ten.

What is the difference between the announcement and effective date?

An index change is published on one day and takes effect on another, usually a few days apart. The effective date answers who was in the index; the announcement answers when the change was public knowledge. Both are true and they are not interchangeable: between them, the two produce different compositions, and a study can easily rank on one while trading the effect of the other.

Can I build a point-in-time universe from periodic member lists?

Partly. Comparing consecutive lists tells you a name arrived or left between two files, but not when. On a monthly file that is a window of thirty days. The honest treatment is to date the change at the later file — never claiming membership earlier than the data supports — and to record the width of the window, so anyone reading the result knows how much of it is inference.

How can I tell whether the file I was given is point-in-time?

Ask whether anything ever left it. A real index over any real history loses names constantly, so a file with no departures at all is a list of current members rather than a history, and it cannot answer a point-in-time question. The second check is whether changes carry two dates; if only one is present, announcement studies on that file are measuring the effective date.

Should index membership be keyed by ticker or by instrument?

By instrument. A ticker is a label the exchange reassigns, and a ticker that names one company today may have named another a few years ago, so a membership history keyed by ticker reintroduces an identity error on top of a survivorship one. Keep the index history in stable identifiers and resolve tickers to them separately.

What about companies that leave the index and later return?

They are two separate spells, not one long membership, and a record that flattens them will show an instrument as a member during a period it was not. Storing each spell as its own interval also makes the gap explicit, which matters because the reason a name left and came back is often exactly the period a strategy is interested in.