AI Development

Backtesting a Simple Sports Prediction Model in Python Without Fooling Yourself

A synthetic Python experiment that shows how a leaky feature turns coin-flip data into a 70% 'model', and how a clean feature and walk-forward testing tell the truth. Education only.

By Rajib Roy Published 5 min read

Sports data is a wonderful place to learn machine learning, and a very good place to fool yourself. Matches have dates, results, teams and statistics, so it is easy to build a feature table in an afternoon and see a number like "70% accuracy" appear. The number feels like a discovery. Often, it is a bug.

This post is about that bug. I built a synthetic dataset where every result is a coin flip, so by construction no honest model can beat chance. Then I built a "model" that scores nearly 70%. Seeing how it happens, and how to catch it, is the most useful thing you can learn before you trust a backtest, in sports or anywhere else. To be clear, this is a lesson in evaluation, not a betting guide.

The setup: a world with no signal

The script creates 3,000 daily matches across 20 teams, with outcomes drawn at random. There is nothing to learn. The best strategy is to always guess the more common outcome, which scored 0.505 on this data. Any method that does significantly better than that is not discovering a pattern, because there isn't one.

Then it builds a feature that looks reasonable: each team's "form", the average of its last five results. This is a classic sports feature. The trick is in how it's computed.

backtest_demo.py

"""Why a sports-prediction backtest can look great and mean nothing.

Everything here is SYNTHETIC: results are random coin flips, so no model can honestly beat ~50%.
Any model that appears to do so is leaking information from the future.
"""
import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
n = 3000
df = pd.DataFrame({
    "date": pd.date_range("2020-01-01", periods=n, freq="D"),
    "team": rng.integers(0, 20, n),
    "win": rng.integers(0, 2, n),          # pure noise
}).sort_values("date").reset_index(drop=True)

# "Form" = a team's average result over its last 5 matches.
g = df.groupby("team")["win"]
df["form_leaky"] = g.transform(lambda s: s.rolling(5, min_periods=1).mean())          # includes today's result!
df["form_clean"] = g.transform(lambda s: s.shift(1).rolling(5, min_periods=1).mean()) # only the past
df = df.dropna().copy()

def accuracy(feature, mask):
    pred = (df.loc[mask, feature] > 0.5).astype(int)
    return (pred == df.loc[mask, "win"]).mean()

split = int(len(df) * 0.7)
train, test = df.index < split, df.index >= split
print(f"matches: {len(df)} | baseline (always guess the majority class): {max(df.win.mean(), 1 - df.win.mean()):.3f}")
print(f"leaky feature, test accuracy: {accuracy('form_leaky', test):.3f}   <- looks like a model, is a bug")
print(f"clean feature, test accuracy: {accuracy('form_clean', test):.3f}   <- honest: about a coin flip")

# Walk-forward evaluation: fit on the past, test on the next block, roll forward.
def walk_forward(feature, block=300):
    scores = []
    for start in range(900, len(df) - block, block):
        test_block = df.iloc[start:start + block]
        scores.append(((test_block[feature] > 0.5).astype(int) == test_block["win"]).mean())
    return np.array(scores)

for f in ["form_clean", "form_leaky"]:
    w = walk_forward(f)
    print(f"walk-forward {f}: mean {w.mean():.3f}, range {w.min():.3f} to {w.max():.3f} over {len(w)} blocks")

Two versions of the same feature

Look at the two transform lines. They differ by one method call:

  • form_leaky uses a rolling mean over the last five results, including the current match. When we predict match number 50, the feature already contains the outcome of match 50.
  • form_clean applies shift(1) first, so each row only sees results that happened before it.

That one line is the difference between knowing the past and knowing the answer. It is also very easy to write by accident, because rolling windows include the current row by default.

What the numbers said

I ran the script and got this output:

matches: 2980 | baseline (always guess the majority class): 0.505
leaky feature, test accuracy: 0.696   <- looks like a model, is a bug
clean feature, test accuracy: 0.499   <- honest: about a coin flip
walk-forward form_clean: mean 0.496, range 0.460 to 0.533 over 6 blocks
walk-forward form_leaky: mean 0.689, range 0.653 to 0.717 over 6 blocks

The leaky feature scored 0.696 on held-out test data, with a proper train/test split by date. The split didn't save it, because the leak was inside the feature itself, not in how I divided the data. The clean version scored 0.499, which is exactly what random data deserves. The scikit-learn documentation defines this failure well: data leakage is using information that would not be available at prediction time, and it produces overly optimistic estimates.

Why a good split is not enough

Beginners are told to split train and test data, and they do. But leakage has many entrances that a split does not close:

  • Features that include the target period, as above. Rolling statistics, season averages and "form" columns are the usual culprits.
  • Random splits on time-ordered data. If you shuffle matches before splitting, the training set contains games from after the ones you test on, so the model effectively sees the future.
  • Preprocessing on the full dataset. Scaling or encoding computed over everything lets test information into training.
  • Information that arrives late. Injury reports, lineups and final statistics may not be known at the time you'd actually make a prediction. If it wasn't available then, it can't be a feature.
  • Selecting a winner on the test set. Try 200 variations, keep the best test score and you've fitted the test set by hand.

Walk-forward evaluation

The honest way to test anything time-ordered is to imitate how you would really use it: train on the past, predict the next block, then roll forward and repeat. scikit-learn ships this idea as TimeSeriesSplit. My script does a minimal version by scoring consecutive 300-match blocks. Two observations from the output:

  • The clean feature's block scores ranged from 0.460 to 0.533 around a mean of 0.496. Even on pure noise, any single block can look better or worse than chance by several points. If I had published only the best block, I could have claimed 53%.
  • The leaky feature stayed between 0.653 and 0.717 in every block. A suspiciously stable, high score is itself a warning. Real predictive edges in noisy domains are small and shaky.

A checklist before you believe a backtest

  1. Write the prediction time. For every feature, state when it becomes known. If you can't, don't use it.
  2. Shift your rolling features. Make "only past rows" a unit test: change a row's result and assert that its own feature does not move.
  3. Split by time, never randomly. Train strictly before test, with a gap if results take time to settle.
  4. Compare with a dumb baseline. Majority class, "home team always wins" or "last season's table". If your model barely beats it, it hasn't learned much.
  5. Report variability, not one number. Score many blocks and show the range.
  6. Run a placebo. Shuffle the labels, or use random data as I did, and check that your pipeline gives chance-level results. If it doesn't, you have a leak.
  7. Count your attempts. The more variations you tried, the more skeptical you should be of the best one.

The placebo test is the best habit here

If I could keep only one habit from this experiment, it would be running the pipeline on data that is random on purpose. It takes a few minutes, needs no domain knowledge and answers the question that matters: can my method produce a good score from nothing? If yes, its good scores on real data mean nothing either.

The same applies far beyond sports: trading strategies, churn models, fraud detection and anything where the future must not leak into the past. A beautiful backtest is not evidence. A backtest that survives a placebo, a time-ordered split and a dumb baseline is a start.

A note on responsibility

Even a well-validated model of a noisy real-world event is uncertain, and results from a past period don't promise the future. Don't treat any model, including one built from this post, as a way to make money, and never stake money you can't afford to lose on a prediction. If gambling is causing you or someone you know problems, a local helpline can help.

Build with me

Need a model you can actually trust?

I build data tools and AI features with proper validation, not just impressive demo numbers. If you want a review of how your model is evaluated, get in touch.

See services Get in touch

What I ran this against

  • Python 3.13 with NumPy and pandas in a virtual environment, run on 17 November 2026. The script is deterministic (fixed random seed).
  • All data is synthetic coin flips. I did not use real matches, real odds or any real team. The point is the method, so the numbers shown say nothing about any real sport or any real prediction model.
  • This is education about evaluating models. It is not betting advice, and nothing here should be used to place a wager.

Sources checked

  • scikit-learn, "Common pitfalls and recommended practices", data leakage section: scikit-learn.org. It defines leakage as information that would not be available at prediction time being used to build the model, producing overly optimistic estimates.
  • scikit-learn, TimeSeriesSplit (train on the past, test on the future): scikit-learn.org.