98% Accuracy? Your Final-Year ML Project Has Data Leakage

You trained the model, printed the score, and got 0.98. The first feeling is relief. The second, if you have been at this long enough, is doubt. If you are searching why is my model accuracy so high, the uncomfortable answer is that a final-year ML project reporting 98% accuracy on a real-world dataset is almost always suffering from data leakage: the test set has already told the model something it was never supposed to know. Leakage is not exotic, and it is not a subtle statistical failure that only researchers hit. It is a few lines of completely ordinary code, written in the order most tutorials teach them, and it will survive every sanity check you know how to run precisely because your accuracy looks great. Leakage does not throw an exception. It hands you a beautiful number and lets you put it on a slide.

This post shows the four leaks that kill student machine learning projects, the wrong code and the right code for each, how to detect them in a dataset you did not create, and the pipeline pattern that makes all four structurally impossible. It also shows, with numbers, where the most popular leak-detection trick on the internet quietly fails.

Every number printed in this post came from actually running the code, on scikit-learn 1.9.0 and imbalanced-learn 0.14.2. The blocks are sequential and share one Python session: names defined in an earlier block are still live in a later one, so run them in order.

What data leakage actually is

Data leakage is any path by which information from your evaluation set reaches your model before evaluation. That is broader than it sounds. It does not require copying rows. A shared mean, a synthetic sample interpolated between a train point and a test point, a second scan of the same patient, a column recorded after the outcome was already known: all of these are paths. The reason it is so destructive is that it inflates exactly the metric you use to decide you are done. A leaking model is not a good model that happened to get lucky on the test split. It is a model whose reported score measures memorisation, and whose real score you have never seen.

Leak 1: you split after SMOTE

This is the leak I see most often, because every imbalanced-data tutorial applies SMOTE to the whole dataframe and only then splits. To show what it is worth on its own, run it against a dataset that contains no signal whatsoever: the features are pure Gaussian noise, the labels are coin flips, there is nothing to learn. Note that the scaler is inside a pipeline and the evaluation is 5-fold cross-validation in both versions below, so the only thing that changes between them is when SMOTE runs.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import StratifiedKFold, cross_val_score
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline

rng = np.random.default_rng(0)
X = rng.normal(size=(400, 40))              # 40 columns of pure noise
y = (rng.random(400) < 0.15).astype(int)    # 15% positives, unrelated to X

cv  = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
clf = RandomForestClassifier(n_estimators=300, random_state=0)

# THE BUG: resample the whole dataset, then cross-validate whatever comes out.
X_res, y_res = SMOTE(random_state=0).fit_resample(X, y)

leaky = Pipeline([("scale", StandardScaler()), ("clf", clf)])
auc = cross_val_score(leaky, X_res, y_res, cv=cv, scoring="roc_auc")
acc = cross_val_score(leaky, X_res, y_res, cv=cv, scoring="accuracy")
print(auc.mean().round(3), auc.std().round(3))   # 0.988 0.007
print(acc.mean().round(3))                       # 0.961

0.988 AUC and 96.1% accuracy on data that contains zero information. This code produces 96% on noise. The mechanism: SMOTE creates synthetic minority points by interpolating between a real minority point and one of its k nearest neighbours. Run it before the split and a synthetic point built from row 12 can land in the validation fold while row 12 itself sits in training. The forest does not generalise. It looks up the answer.

honest = ImbPipeline([
    ("scale", StandardScaler()),
    ("smote", SMOTE(random_state=0)),   # now resampling happens inside each fold
    ("clf",   clf),
])

auc = cross_val_score(honest, X, y, cv=cv, scoring="roc_auc")
print(auc.mean().round(3), auc.std().round(3))   # 0.533 0.063

0.533. Chance, which is the correct answer for noise. Same data, same model, same SMOTE, same scaler, same folds, same seed. The only thing that moved is when the resampling happened, and it is worth 0.45 AUC of pure fiction.

Leak 2: you fit the scaler on the full dataset

Quieter, and common in published notebooks. The leak is that fit_transform on the whole matrix computes the mean and standard deviation of every column using the test rows, then bakes those statistics into the training data. Your model is trained on a representation that already knows the test set's distribution.

from sklearn.model_selection import train_test_split

# WRONG: the scaler sees every row before the split
scaler = StandardScaler()
X_all  = scaler.fit_transform(X)          # mean and std computed over test rows too
X_tr, X_te, y_tr, y_te = train_test_split(X_all, y, test_size=0.2, random_state=0)

# RIGHT: fit on train, transform both
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=0)
scaler = StandardScaler().fit(X_tr)       # fit on train only
X_tr   = scaler.transform(X_tr)
X_te   = scaler.transform(X_te)           # transform. never fit.

With a plain StandardScaler this leak is usually worth a point or two of accuracy, not thirty. It is dangerous because it rarely arrives alone, and because the same mistake with a transformer that looks at y is devastating. Fit SelectKBest on all 400 rows of a 5000-column noise matrix and the resulting model scores above 0.8 AUC on noise, by itself. The rule has no exceptions: a transformer may only ever call fit on training data. Anything that learns from data and runs before the split is a leak.

• SelectKBest or any feature selection fitted on all rows. This one is brutal, because the selector reads the labels.
• Target or mean encoding of a categorical column computed over the whole dataframe. This encodes the label directly.
• SimpleImputer or KNNImputer filling missing values using the global median.
• PCA or any dimensionality reduction fitted before the split.
• TfidfVectorizer fitted on train plus test documents. The IDF weights are learned from the corpus.
• Any manual df.col.fillna(df.col.mean()) written before the split. Same bug, no library to blame.

Leak 3: you split rows when you should have split patients

This is the leak that ruins medical imaging, audio, sensor and video projects, and it is the one students defend hardest, because the split looks correct. You have 1920 visits from 240 patients, eight visits each. You call train_test_split. No row appears in both sets. It still leaks, because the model does not need the same row twice. It only needs another row from the same person. Here is a cohort where age, bmi, sex and site are patient attributes, so they repeat identically across that patient's eight visits, and where the outcome is a genuine but noisy function of them.

import pandas as pd

rng = np.random.default_rng(1)
n_pat, per_pat = 240, 8
n   = n_pat * per_pat
pid = np.repeat(np.arange(n_pat), per_pat)      # 8 visits per patient

age_p  = rng.normal(58, 12, n_pat).round(0)     # patient-level attributes:
bmi_p  = rng.normal(27, 4.5, n_pat).round(1)    # constant across that patient's visits
sbp_p  = rng.normal(130, 15, n_pat)
sex_p  = rng.choice(["F", "M"], n_pat)
site_p = rng.choice(["A", "B", "C"], n_pat)

# the outcome is a real, noisy function of patient-level risk
logit   = -2.0 + 0.08*(age_p-58) + 0.26*(bmi_p-27) + 1.0*(sex_p=="M") + 0.05*(sbp_p-130)
label_p = (rng.random(n_pat) < 1/(1+np.exp(-logit))).astype(int)

y  = pd.Series(label_p[pid], name="relapse")   # the label is a property of the PATIENT
df = pd.DataFrame({
    "patient_id": pid,
    "age":  age_p[pid],
    "bmi":  bmi_p[pid],
    "sbp": (sbp_p[pid] + rng.normal(0, 6, n)).round(1),   # varies per visit
    "sex":  sex_p[pid],
    "site": site_p[pid],
})
# a follow-up is scheduled BECAUSE of the outcome (we will come back to this)
df["days_to_followup"] = np.where(y == 1, rng.normal(10, 2, n), rng.normal(88, 6, n)).round(1)
df["relapse_note_len"] = np.where(y == 1, rng.normal(240, 40, n).round(0), np.nan)
df.loc[rng.random(n) < 0.05, "bmi"] = np.nan   # realistic missingness

print(len(df), df["patient_id"].nunique(), round(y.mean(), 3))   # 1920 240 0.221

Because a patient's age and bmi are identical on all eight of their visits, those columns are a fingerprint. A tree only has to recognise the person to recover their label. That is exactly what an augmented crop, a second slice, or a five-second window of the same recording gives a model in real data.

from sklearn.model_selection import StratifiedGroupKFold
from sklearn.metrics import roc_auc_score

Xd = pd.get_dummies(df[["age", "bmi", "sbp", "sex", "site"]], columns=["sex", "site"])
Xd["bmi"] = Xd["bmi"].fillna(Xd["bmi"].median())

# WRONG: random row split. the same patient lands in train AND in test.
X_tr, X_te, y_tr, y_te = train_test_split(Xd, y, test_size=0.2, stratify=y, random_state=0)
m = RandomForestClassifier(n_estimators=300, random_state=0).fit(X_tr, y_tr)
print(round(roc_auc_score(y_te, m.predict_proba(X_te)[:, 1]), 3))   # 0.997

# RIGHT: the patient is the unit. no patient may cross the boundary.
sgkf = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)
s = cross_val_score(RandomForestClassifier(n_estimators=300, random_state=0),
                    Xd, y, groups=df["patient_id"], cv=sgkf, scoring="roc_auc")
print(s.mean().round(3), s.std().round(3))                          # 0.688 0.119

0.997 against 0.688. The signal in this cohort is real, so the honest number is not chance, it is just far lower than the leak suggested. That is the shape of the damage: leakage does not invent a model out of nothing, it inflates a mediocre one into a triumphant one. Use GroupShuffleSplit for a single held-out split, StratifiedGroupKFold when you also need class balance preserved, and pass the group vector through the groups argument every single time.

Leak 4: a column that already contains the answer

Target leakage is when a feature was recorded after, or because of, the outcome you are predicting. Predicting readmission with a discharge_summary_length column. Predicting default with a recovery_agent_assigned flag. Predicting churn with a cancellation_reason that is null for everyone who stayed. The model is not wrong to use it, your dataset is wrong to contain it. You will not find this on a correlation heatmap. Score every numeric column against the label on its own and read the ranking: a single raw column that separates the classes almost perfectly is not a feature, it is a confession.

def leak_scan(X: pd.DataFrame, y: pd.Series) -> pd.Series:
    """AUC of every numeric column, on its own, against the target."""
    scores = {}
    for col in X.select_dtypes(include="number").columns:
        v  = X[col]
        ok = v.notna()
        # a column that is null for one whole class leaves a single class here,
        # and roc_auc_score raises ValueError. that column is the loudest leak
        # in the table, so record it rather than crashing on it.
        if ok.sum() == 0 or y[ok].nunique() < 2:
            scores[col] = np.nan
            continue
        auc = roc_auc_score(y[ok], v[ok])
        scores[col] = max(auc, 1 - auc)   # a column can predict either direction
    return pd.Series(scores).sort_values(ascending=False)

print(leak_scan(df.drop(columns=["patient_id"]), y).round(3).to_string())
# days_to_followup    1.000    <- the label in disguise
# bmi                 0.757    <- a real predictor. this is what signal looks like
# sbp                 0.667
# age                 0.639
# relapse_note_len      NaN    <- null for every negative. guarded, not scored

Read the two extremes. days_to_followup scores a perfect 1.000, because the follow-up was scheduled as a consequence of the relapse. relapse_note_len cannot be scored at all: it is null for every patient who did not relapse, so once you drop the nulls only one class survives. The naive version of this function raises ValueError there, which means the check breaks on the most blatant leak in the table. And bmi at 0.757 is the control: a genuinely predictive column, high but not absurd. Any column above roughly 0.90 deserves an interrogation, and the question is always the same. At the moment I would have to make this prediction in production, would this value exist yet? If not, drop the column. Do not scale it, do not regularise it away, drop it.

The pattern that makes all four leaks impossible

Do not fix leaks one at a time. Restructure the code so they cannot be expressed. Every fit-based step goes inside one estimator, that estimator is the only object that ever touches data, and the splitter is group-aware. This is the template I would actually write.

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OrdinalEncoder
from sklearn.model_selection import cross_validate
from imblearn.over_sampling import SMOTENC

num = ["age", "bmi", "sbp"]
cat = ["sex", "site"]
# note: days_to_followup and relapse_note_len are deliberately NOT here.

prep = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale",  StandardScaler())]), num),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("ord",    OrdinalEncoder(handle_unknown="use_encoded_value",
                                                unknown_value=-1))]), cat),
])
cat_idx = list(range(len(num), len(num) + len(cat)))   # [3, 4] after the transformer

pipe = ImbPipeline([
    ("prep",  prep),                                            # fitted per fold
    ("smote", SMOTENC(categorical_features=cat_idx, random_state=0)),  # training fold only
    ("clf",   RandomForestClassifier(n_estimators=300, random_state=0)),
])

cv  = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=0)
res = cross_validate(pipe, df, y, groups=df["patient_id"], cv=cv,
                     scoring=["roc_auc", "average_precision", "balanced_accuracy"],
                     return_train_score=True)

print(res["test_roc_auc"].mean().round(3), res["test_roc_auc"].std().round(3))  # 0.691 0.123
print(res["train_roc_auc"].mean().round(3))                                     # 1.0
print(res["test_average_precision"].mean().round(3))                            # 0.385
print(res["test_balanced_accuracy"].mean().round(3))                            # 0.569

Note what this buys. The imputer's median is recomputed on four folds and applied to the fifth. The scaler never sees a validation row. The sampler runs on the training fold and is skipped at predict time. No patient crosses a fold boundary. And return_train_score hands you the gap for free: a train AUC of 1.0 against a test AUC of 0.691 is a forest memorising its training fold, which is overfitting rather than leakage, but it is a different disease worth seeing on the same line.

Three checks to run before you believe any number

1. Shuffle the labels, at the level the label actually lives at

Permute y, keep X, rerun the exact evaluation you plan to report. A correct pipeline must collapse to chance. Here is the part almost every version of this advice gets wrong: if the label is a property of the group, you must permute it at the group level. Shuffling row by row breaks the within-patient constancy of the label, which destroys the very structure the group leak exploits, and the leaky pipeline then reports a clean 0.5 while still being catastrophically broken. Take one label per patient, permute those, broadcast back.

def group_shuffle_null(pipe, df, y, groups, cv, n_repeats=10):
    """Permute the label at the GROUP level, then re-run the evaluation."""
    lab = y.groupby(groups).first()          # one label per patient
    out = []
    for seed in range(n_repeats):
        r      = np.random.default_rng(seed)
        perm   = pd.Series(r.permutation(lab.to_numpy()), index=lab.index)
        y_null = groups.map(perm)            # broadcast back to all 8 visits
        out.append(cross_val_score(pipe, df, y_null, groups=groups,
                                   cv=cv, scoring="roc_auc").mean())
    return np.array(out)

null = group_shuffle_null(pipe, df, y, df["patient_id"], cv)
print(null.mean().round(3), null.std().round(3))   # 0.505 0.039  <- correct pipeline

# now feed the SAME group-permuted labels to the LEAKY row-split evaluation
lab    = y.groupby(df["patient_id"]).first()
r      = np.random.default_rng(0)
y_null = df["patient_id"].map(pd.Series(r.permutation(lab.to_numpy()), index=lab.index))

a, b, c, d = train_test_split(Xd, y_null, test_size=0.2, stratify=y_null, random_state=0)
mm = RandomForestClassifier(n_estimators=300, random_state=0).fit(a, c)
print(round(roc_auc_score(d, mm.predict_proba(b)[:, 1]), 3))   # 0.984  <- leak exposed

# and the row-wise shuffle everyone recommends, on the same leaky evaluation:
r       = np.random.default_rng(0)
y_rowsh = pd.Series(r.permutation(y.to_numpy()), index=y.index)
a, b, c, d = train_test_split(Xd, y_rowsh, test_size=0.2, stratify=y_rowsh, random_state=0)
mm = RandomForestClassifier(n_estimators=300, random_state=0).fit(a, c)
print(round(roc_auc_score(d, mm.predict_proba(b)[:, 1]), 3))   # 0.513  <- leak INVISIBLE

0.505 for the correct pipeline, which is what a passing check looks like. 0.984 for the leaky row split under a group-level permutation, which is what a failing one looks like. And 0.513 for that same broken pipeline under the row-wise shuffle: a clean bill of health for a model that scores a fraudulent 0.997 on real labels. Repeat the permutation a handful of times and read the mean, because a single draw over 240 patients has enough variance to land near 0.59 by luck alone. Run this check with the same splitter and groups you intend to report.

2. Count duplicate rows

Call df.duplicated().sum() before you split. The cohort above returns 0, which is the answer you want. Public student datasets, especially the ones circulated on Kaggle and in college repos, are frequently pre-balanced by someone who duplicated the minority class. If they did, a random split puts identical rows on both sides and your accuracy is a lookup table. Deduplicate first, then split, and find out how the file was balanced before you trust it.

3. Assert the group boundary

Do not trust that your splitter did what you think. Assert it, and leave the assert in the code so the examiner can see it. This is the check that catches Leak 3, and it is three lines.

from sklearn.model_selection import GroupShuffleSplit

g   = df["patient_id"].to_numpy()
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
tr, te = next(gss.split(df, y, groups=g))

assert not (set(g[tr]) & set(g[te])), "patient in both sets"

What the examiner asks, and what an honest number looks like

1. 1Show me the line where you split and the line where you resampled. Which one runs first?
2. 2What is your class balance? If it is 95/5, what does 96% accuracy prove?
3. 3Is your test set a random sample of rows, or of patients, users, sessions, or days?
4. 4Which feature has the highest importance, and when in the real timeline is that value recorded?
5. 5What happens if I shuffle the labels? At what level did you shuffle them?
6. 6What is the variance across your cross-validation folds? One number from one split tells me nothing.
7. 7Was the dataset already balanced when you downloaded it, and if so, how?

None of these require reading your model code. Leakage lives in the plumbing, and the plumbing is where an experienced reviewer looks first. So report AUC or average precision rather than accuracy on any imbalanced problem, and report the spread across folds. A mean of 0.69 with a standard deviation of 0.12 across five group-aware folds is a defensible result you can talk about. A single 0.98 from one train_test_split is not a result at all, it is an anecdote, and it is usually an anecdote about your preprocessing order.

Fixing this makes your project stronger, not weaker. A project that reports 0.69 honestly, shows the leak it found, and demonstrates the shuffled-label control is a substantially better piece of engineering than one that reports 0.98 and cannot survive a follow-up question. The examiner is not grading the number, they are grading whether you know what the number means. That distinction, between a metric that looks good and a system that works, is the first thing we look for when we review a student ML project at Tenzok, and it is worth building the habit long before anyone is grading you for it. Start by running the group-level shuffle on whatever you trained last week.