Backtest Validation: Catching Lookahead Bias

Resumen

Pixel’s announcement explores a four-layer approach to validating trading strategies after a backtest passed three statistical checks but contained lookahead bias. It explains what the Deflated Sharpe Ratio, Monte Carlo ruin tests, split-half stability, and a fourth audit layer can catch—and why honest trade generation matters.

DEV Community

Three-Layer Strategy Validation and the Blind Spot That Bit Me

Three-Layer Strategy Validation and the Blind Spot That Bit Me My validation stack had three layers, and all three were green. DSR came out at 1.000. Coin-block Monte Carlo showed a 0% ruin rate across hundreds of randomized pool combinations. Split-half stability was positive in both halves of the trade sequence. I shipped it, and the strategy was garbage, because the backtest that produced those numbers had a lookahead bug that leaked future candle data into the entry decision. The uncomfortable lesson: all three layers evaluate the statistical properties of one trade sequence. They assume that sequence was generated honestly. None of them audits how trades were generated. A poisoned sequence can be perfectly well-behaved statistically — in fact, a lookahead bug usually makes it more well-behaved, because the strategy is effectively peeking at the answer. High DSR, zero ruin, clean split-half — that's exactly what a leak looks like. This post is the four-layer design I run now, with the code signatures and a table of what each layer actually catches. The fourth layer is the one that would have saved me. Layer 1: Deflated Sharpe Ratio The DSR (Bailey & López de Prado) adjusts an observed Sharpe ratio for the number of trials you ran, and for the non-normality of returns. If you tried 40 parameter sets and kept the best, the DSR asks: given 40 trials, skew, and kurtosis, is this Sharpe still distinguishable from the maximum you'd expect from noise? import numpy as np
from scipy.stats import norm

def deflated_sharpe_ratio(
returns: np.ndarray,
n_trials: int,
sr_variance: float | None = None,
periods_per_year: int = 252,
) -> float:
"""
Returns the probability that the true Sharpe exceeds zero,
after deflating for n_trials and return non-normality.

returns: per-period (e.g. daily) strategy returns, chronological
n_trials: number of configurations you actually evaluated
sr_variance: variance of Sharpe across your trials; if None,
             use the estimator below (conservative for small n_trials)
"""
r = np.asarray(returns, dtype=float)
r = r[np.isfinite(r)]
n = r.size
if n < 30 or n_trials < 1:
    return 0.0

mu, sd = r.mean(), r.std(ddof=1)
if sd == 0:
    return 0.0
sr = mu / sd                      # per-period Sharpe
g1 = ((r - mu) ** 3).mean() / sd**3
g2 = ((r - mu) ** 4).mean() / sd**4

# Variance of the Sharpe estimator under skew/kurtosis
sr_var = (1 - g1 * sr + (g2 - 1) / 4 * sr**2) / (n - 1)

if sr_variance is None:
    # Expected max of n_trials draws from N(0, sr_var)
    e_max = np.sqrt(sr_var) * (
        (1 - np.euler_gamma) * norm.ppf(1 - 1 / n_trials)
        + np.euler_gamma * norm.ppf(1 - 1 / (n_trials * np.e))
    )
else:
    e_max = np.sqrt(sr_variance) * norm.ppf(1 - 1 / n_trials)

z = (sr - e_max) / np.sqrt(sr_var)
return float(norm.cdf(z))

Two things matter operationally. First, n_trials must be honest — every grid point, every filter variant, every "quick check" counts. Second, the DSR is only as good as the return series you feed it. It has no opinion on whether that series is real. Layer 2: Coin-Block Monte Carlo I don't bootstrap individual trades. Trades cluster — volatility regimes, correlated entries, streaks. So I resample in blocks with random block boundaries, and I randomize the pool composition across hundreds of runs: which assets or sub-strategies are included, in what weights, drawn from the eligible universe. The output I care about is the ruin rate — the fraction of simulated paths that breach a drawdown or capital threshold. def coin_block_monte_carlo(
trade_pnls: np.ndarray,
n_sims: int = 500,
mean_block: int = 10,
ruin_dd: float = 0.25,
initial_capital: float = 1.0,
rng: np.random.Generator | None = None,
) -> dict:
"""
Resample the trade sequence in random-length blocks and measure
the fraction of pat...
Backtest Validation: Catching Lookahead Bias
Powered by Readybot.io

Lo último de Pixel

Stop Paying for AI Renders in Test Orders

Pixel’s announcement explains how internal test orders were accidentally triggering paid AI video renders costing just over $1 each. It details a server-side dry-run guard …

Sandblasting Grit Guide: 120 vs 220 vs 320

Pixel’s announcement explains how 120, 220, and 320 grit affect aluminium’s matte finish and reveal die-cast porosity. It also covers how pressure, blast distance, and …

NVIDIA Cosmos 3 Edge: Physical AI on Jetson

Pixel shares the first article in CAVEDU's Cosmos series, introducing NVIDIA Cosmos 3 Edge, an approximately 4-billion-parameter world model for Jetson devices. It explores multimodal …

Laravel Semantic Search with OpenAI and Pinecone

Pixel shares a practical guide to building semantic search in Laravel using OpenAI embeddings and Pinecone. It covers 1536-dimensional vectors, cosine similarity, index setup, environment …