cryptotradingbook

Backtesting pitfalls

Foundations → Intermediate · ~12 min read

At a glance

  • A backtest is a historical anecdote with math - not a forecast.
  • Curve fitting: enough free parameters fit any history.
  • Survivorship and selection bias: dead coins and dead exchanges are missing from easy data.
  • Execution costs (fees, slippage, funding) turn many paper edges into live losses.
  • Out-of-sample testing is the minimum bar; walk-forward is the honest one.

1. What a backtest can honestly show

Whether a rule set was mechanically consistent, whether its risk sizing would have survived known regimes, and whether it behaves sanely in extreme data (liquidity gaps, flash crashes). That is worth a lot - and it is all. The future is not in the file.

2. The five ways backtests lie

Curve fitting (too many parameters). Survivorship bias: today's coin list excludes everything delisted or dead - including the exact strategies that failed (survivorship bias). Look-ahead bias: using data that was only known later (adjusted closes, delayed prints). Execution costs ignored: fees, slippage, funding. Regime concentration: a 2019-2022 test covers two regimes at most - one bull, one catastrophe, and nothing between.

WHAT YOU TESTED✓ Trend system 2020-2025✓ Breakout system v2✓ Mean reversion (refined)the one strategy you refinedWHAT YOU FORGOT✗ System #1: blown account✗ System #2: abandoned✗ System #3: overfit✗ System #4: …20 more⇄Testing three variations of the idea that survived is survivorship bias -the seven you abandoned were the actual sample.
Figure 12.2 — In backtesting, survivorship bias hides your own dead strategies: the rule set that "works" is the one that survived your last seven attempts.

3. The minimum honest protocol

Fix the rules first, freeze them, then test on data you did not use to design them (out-of-sample). Include realistic fees (taker, both sides), a slippage model, and the funding rate for any perp strategy. Test across at least three regimes including a drawdown period. Report the loss streaks, not just the total return - the loss streak is what actually tests your discipline.

Remember: If you would not trade the backtest's worst drawdown at its reported size, you would not have traded the backtest. The worst drawdown is the deliverable, not the CAGR.

4. Paper trading's honest place

Paper trading tests execution mechanics (order types, API flow, fills) - not psychology. Use it to verify the plumbing after the backtest and before small live size. Do not use its P&L as evidence of an edge: courage is the missing ingredient and it cannot be simulated.

5. When this fails

Even a clean protocol fails when the market regime ends or when the strategy is a good description of one asset's past decade. The final test is live size: smallest meaningful position, live slippage, real emotions - and the journal comparing plan vs. execution from day one.

Risk: The most dangerous number in trading is a confident backtest. It authorises size. Size is what turns ordinary statistical bad luck into permanent loss.

Knowledge check

  1. What can a backtest honestly demonstrate? (Mechanical consistency and survivability in known regimes - not the future.)
  2. Name three of the five classic backtest lies. (Curve fitting, survivorship bias, look-ahead bias, ignored execution costs, regime concentration.)
  3. Why must loss streaks be reported, not just returns? (They are what tests discipline and risk sizing in reality.)
  4. What is paper trading's honest use? (Testing execution mechanics - not psychology, not edge evidence.)
  5. Which backtest number authorises size - and why is it dangerous? (The confident backtest result; it licenses size that converts bad luck into permanent loss.)