Backtest Validation: Catching Lookahead Bias
Resumen
Pixel’s announcement explores a four-layer approach to validating trading strategies after a backtest passed three statistical checks but contained lookahead bias. It explains what the Deflated Sharpe Ratio, Monte Carlo ruin tests, split-half stability, and a fourth audit layer can catch—and why honest trade generation matters.
DEV Community
Three-Layer Strategy Validation and the Blind Spot That Bit Me
Three-Layer Strategy Validation and the Blind Spot That Bit Me My validation stack had three layers, and all three were green. DSR came out at 1.000. Coin-block Monte Carlo showed a 0% ruin rate across hundreds of randomized pool combinations. Split-half stability was positive in both halves of the trade sequence. I shipped it, and the strategy was garbage, because the backtest that produced those numbers had a lookahead bug that leaked future candle data into the entry decision. The uncomfortable lesson: all three layers evaluate the statistical properties of one trade sequence. They assume that sequence was generated honestly. None of them audits how trades were generated. A poisoned sequence can be perfectly well-behaved statistically — in fact, a lookahead bug usually makes it more well-behaved, because the strategy is effectively peeking at the answer. High DSR, zero ruin, clean split-half — that's exactly what a leak looks like. This post is the four-layer design I run now, with the code signatures and a table of what each layer actually catches. The fourth layer is the one that would have saved me. Layer 1: Deflated Sharpe Ratio The DSR (Bailey & López de Prado) adjusts an observed Sharpe ratio for the number of trials you ran, and for the non-normality of returns. If you tried 40 parameter sets and kept the best, the DSR asks: given 40 trials, skew, and kurtosis, is this Sharpe still distinguishable from the maximum you'd expect from noise? import numpy as np
from scipy.stats import norm
def deflated_sharpe_ratio(
returns: np.ndarray,
n_trials: int,
sr_variance: float | None = None,
periods_per_year: int = 252,
) -> float:
"""
Returns the probability that the true Sharpe exceeds zero,
after deflating for n_trials and return non-normality.
returns: per-period (e.g. daily) strategy returns, chronological
n_trials: number of configurations you actually evaluated
sr_variance: variance of Sharpe across your trials; if None,
use the estimator below (conservative for small n_trials)
"""
r = np.asarray(returns, dtype=float)
r = r[np.isfinite(r)]
n = r.size
if n < 30 or n_trials < 1:
return 0.0
mu, sd = r.mean(), r.std(ddof=1)
if sd == 0:
return 0.0
sr = mu / sd # per-period Sharpe
g1 = ((r - mu) ** 3).mean() / sd**3
g2 = ((r - mu) ** 4).mean() / sd**4
# Variance of the Sharpe estimator under skew/kurtosis
sr_var = (1 - g1 * sr + (g2 - 1) / 4 * sr**2) / (n - 1)
if sr_variance is None:
# Expected max of n_trials draws from N(0, sr_var)
e_max = np.sqrt(sr_var) * (
(1 - np.euler_gamma) * norm.ppf(1 - 1 / n_trials)
+ np.euler_gamma * norm.ppf(1 - 1 / (n_trials * np.e))
)
else:
e_max = np.sqrt(sr_variance) * norm.ppf(1 - 1 / n_trials)
z = (sr - e_max) / np.sqrt(sr_var)
return float(norm.cdf(z))
Two things matter operationally. First, n_trials must be honest — every grid point, every filter variant, every "quick check" counts. Second, the DSR is only as good as the return series you feed it. It has no opinion on whether that series is real. Layer 2: Coin-Block Monte Carlo I don't bootstrap individual trades. Trades cluster — volatility regimes, correlated entries, streaks. So I resample in blocks with random block boundaries, and I randomize the pool composition across hundreds of runs: which assets or sub-strategies are included, in what weights, drawn from the eligible universe. The output I care about is the ruin rate — the fraction of simulated paths that breach a drawdown or capital threshold. def coin_block_monte_carlo(
trade_pnls: np.ndarray,
n_sims: int = 500,
mean_block: int = 10,
ruin_dd: float = 0.25,
initial_capital: float = 1.0,
rng: np.random.Generator | None = None,
) -> dict:
"""
Resample the trade sequence in random-length blocks and measure
the fraction of pat...

Powered by Readybot.io