What a backtest can — and can't — tell you

You ran a backtest. The equity curve goes up and to the right. Before you do anything else, work out how much of that number is your strategy and how much is the way you produced it.

A backtest answers exactly one question well: what would these rules have done over this data? Every other question you want to ask it — will this work, is it better than that one, should I use it — is an inference you are making on top, and each of the traps below is a way that inference goes wrong.

Overfitting

You try 10 and 20. Then 5 and 50. Then 12 and 26. The third one looks best, so you keep it.

You have not found a better strategy. You have found which parameters happened to fit the noise in the window you tested, and you have used up your data doing it. The more combinations you try, the more certain it is that the best-looking result is the luckiest one rather than the soundest one.

The tell is sensitivity. If 12/26 returns 18% and 11/26 returns 2%, you have not found an edge — you have found a coincidence sitting on a cliff edge. A parameter that matters should degrade gently as you move it.

Lookahead bias

Using information the strategy could not have had at the moment it acted.

It creeps in quietly. Deciding on a bar's close and filling at that same close. Using an indicator computed over the full series including the future. Filtering your symbol list by something you only know now — like which companies still exist.

CodeBull's engine calls on_new_tick with one price at a time and gives the strategy no way to see forward, which removes the most common form. It does not remove the version where you looked at the whole chart, noticed the crash in the middle, and wrote a rule that avoids it.

Survivorship

Test a strategy on today's index members over the last ten years and you have quietly restricted yourself to companies that made it. The ones that were delisted, acquired or wiped out are not in your data — and they are exactly where a strategy would have lost.

Costs

Spreads, fees and slippage are modelled in a simulation, and a model is a simplification. A strategy that trades often is far more sensitive to how those costs are modelled than to its own logic: halve the assumed cost and a losing high-frequency rule turns profitable on paper without a single line changing.

Check the trade count first. A strategy making four hundred trades is a bet on the cost model. One making eleven is a bet on eleven decisions.

Regime

Every backtest is a sample of one market. A trend-following rule tested across 2020–2021 looks brilliant, because that period trended. The same rule across a sideways stretch grinds down. Neither result is the strategy's "true" performance — both are the strategy meeting one particular regime.

Run the same strategy over several disjoint windows. If the answer changes completely between them, the answer you liked was a property of the window.

What survives all of this

A backtest is still the cheapest way to find out that your logic is wrong, and that remains its highest-value use. If the trade list does not match what you intended, nothing else about the run matters. Read the trades before you read the return.

Beyond that, treat a backtest as evidence with a large error bar, not a measurement. Simple rules studied carefully — as in Brock, Lakonishok and LeBaron's 1992 study of technical trading rules — come with pages of qualification about data-snooping and transaction costs precisely because the naive reading of the result is the wrong one. Their caution is the point, not the headline number.

A checklist

Before you trust a backtest at all:

• Does the trade list match what you intended? Check five trades by hand.
• How many parameter combinations did you try before this one?
• Does the result survive moving each parameter one step?
• Does it survive a different, non-overlapping time window?
• How many trades did it make, and how sensitive is the result to costs?
• Is there anything in the rules you only knew because you had seen the chart?

If you have not run your first strategy bot yet, start there — the checklist is easier to apply to a strategy you wrote than one you read about. And paper trading, explained covers the wider question of what a simulator can and cannot show you.

Educational content, not financial advice. Backtest results on CodeBull are produced on simulated money and do not predict real returns. No real-money trading takes place on this platform.