Why your backtest doesn't match live trading

Every systematic team has had this moment. The backtest looked clean. Walk-forward held up. The strategy was promoted, and three weeks later the numbers in the account don't match the numbers in the model. Not by a lot. Just enough that neither can be trusted.

What follows is the list of places the gap usually comes from, roughly in the order we'd check them, then a worked example, then a description of the operating process that stops the gap opening in the first place. None of this is exotic. Most of it is well known. The problem is that in most shops it lives in different tools, different scripts and different people's heads, and the glue between them is where the truth leaks out.

1. The cost model was optimistic

The backtest charged a flat commission and assumed fills at the close. The live strategy paid commission, exchange fees, the bid-ask spread, and market impact on the days it mattered, which are exactly the volatile days the strategy was designed to trade.

A cost model has to cover at least four things: commission and fees per venue, the spread at the time of day the strategy actually trades, an impact term that grows with order size relative to typical volume, and a realistic fill assumption. "Fill at the close" is a modelling convenience, not something a broker offers. If the backtest was run once with costs switched off "to see the raw signal" and never run again with them on, that alone can account for the whole gap.

2. Fills didn't happen the way the model assumed

The model assumed every order filled in full at the signal price. Live, some orders filled partially, some were rejected, some sat as limit orders that never traded, and some filled at a worse price after a fast move. Each of those leaves the live book holding a different position from the model book, and the difference compounds through every later decision the strategy makes from that position.

This one is invisible unless the system compares what it asked the broker to do with what the broker did, order by order, every day.

3. Look-ahead crept in

The research pipeline used a data field that isn't available at the moment of the decision. The classic cases: a daily bar's close used to trade "on" that bar rather than the next; fundamentals stamped with the period they describe rather than the date they were published; index membership taken from today's list rather than the list as it stood at the time. Each is a small piece of future information. Together they make a backtest look considerably better than anything a live process can reproduce.

The test for look-ahead is mechanical: at every decision timestamp, could the process have had this value? If the data platform can't answer that question per field, the honest answer is "probably not always".

4. The data changed after the backtest ran

Prices get corrected. Corporate actions get applied late. A vendor restates a series. A symbol is remapped. The backtest ran against the dataset as it was in March; the live strategy runs against the dataset as it is now, and the two are not the same file.

Data drift is why the question "can you re-run last quarter's backtest and get the same number?" matters so much. If the answer is no, the backtest can't be a baseline for anything, because nobody knows what changed: the data, the code or the parameters.

5. Calendars and clocks were wrong somewhere

Timezones, daylight-saving changes, exchange holidays, half days, and the difference between the data vendor's session times and the broker's. A strategy that trades the open in a backtest and trades an hour after the open in production, because a server sits in a different timezone from the data, is trading a different strategy.

6. The parameters drifted between research and production

The research notebook had the lookback at 60. The production config had it at 50, because someone tried something in June and never changed it back. Or the production code was updated to fix a bug, and the backtest was never re-run on the fixed code. Or the strategy was tuned on one version of the signal library and deployed on another.

None of these is anyone's fault. They are what happens when the strategy, its parameters, the data it uses and the code it runs on are not versioned together as one thing that can be named, compared and promoted.

7. The operational gap: what the broker actually did

This is the one that swallows the others. The model believes it holds a position. The broker's statement says something slightly different, because of a partial fill on Tuesday, a rejected order on Thursday that nobody saw, a corporate action the model didn't apply, and a fee the model didn't book. Each is small. The model and the account have been slowly diverging for weeks, and every decision since has been made from a position that isn't real.

The only fix is to treat the broker's record as the source of truth, reconcile the model against it every day, and stop when they disagree.

A worked example

Take a medium-frequency trend strategy on a basket of liquid futures, rebalancing daily at the close. In research it was validated with walk-forward analysis over several years and looked robust across a neighbourhood of parameters. It was promoted to paper, then live.

By the end of the first month the live book was noticeably behind the model. Working through the list:

• Costs: the backtest used a flat commission and closing-price fills. The live process paid the spread on every rebalance and, on the two or three most volatile days, moved the market against itself. That was roughly half the gap.
• Fills: on one high-volume day, two of the basket's orders filled partially; the model assumed full fills, so its positions in those two contracts were wrong from then on.
• Calendar: one contract's roll date differed between the vendor's continuous series and the broker's actual expiry, so the model rolled a day late.
• Parameters: the production config had a slightly different volatility window from the research run, left over from an earlier experiment.

Nobody had done anything unreasonable. Each cause was found in a different tool, by a different person, over several days. And because the original backtest couldn't be re-run bit-for-bit, the team couldn't say with confidence that they had found everything; they could only say the gap looked smaller now.

What a process that catches this looks like

The pattern is the same whether it's built in-house or bought, and it has four properties.

Everything is versioned together. A strategy version names the code, the parameters, the data snapshot and the environment it was validated in. Promoting to paper or live means promoting that exact version, not a config file that someone edits.

Every run leaves reproducible artifacts. A backtest produces stored inputs and outputs that can be re-run later to the same result, so a live run always has a baseline to be compared against, and a drift can be traced to data, code or parameters.

Promotion is an explicit step. Nothing goes to paper or live without validation checks (walk-forward, out-of-sample, cost model, parameter stability) having been run on that version and someone approving the result. The approval is recorded.

Operations start from broker truth. Positions, fills, fees and cash are pulled from the broker and reconciled against the model every day. A disagreement is a stop, not a note in a spreadsheet.

Institutional desks have most of this because they have the engineering teams to build it. Smaller systematic teams, proprietary firms and emerging managers usually have parts of it, held together by scripts and habit, and the gap between backtest and live is where that shows.

Where Vanadyr fits

This is the problem Vanadyr OS is built around: one system in which strategies are versioned, runs leave reproducible artifacts, promotion goes through validation and an explicit approval, and operations reconcile daily against what the broker actually holds. It is in private development, and we are preparing a limited cohort of design partners, teams who live this problem, to shape what ships first.