Building FeatureSmith: code-review discipline for ML datasets

Datasets skip the review that code gets

In software, a pull request runs through linters, type checkers, and tests before anyone merges it. Broken code rarely reaches production unseen.

Datasets get no such step. A CSV lands in a folder and goes straight into a training run. Target leakage, hidden missingness, duplicated rows, and schema changes stay invisible until a model fails in validation, or worse, in production. By then the GPU hours are spent.

FeatureSmith is my answer: a reviewer for tabular ML datasets, in the spirit of Ruff or ESLint. Point it at a dataset and a target column, and it tells you what is wrong before you train.

What target leakage looks like

Leakage happens when a feature carries information you will not have at prediction time. Common shapes:

• A timestamp recorded after the event you are trying to predict.
• An ID or status column that quietly encodes the label.
• A feature so correlated with the target that it is effectively the answer.

A leaky model scores beautifully in training and falls apart on real data. The fix is cheap if you catch it early, which is the whole point of reviewing the dataset first.

Ten reviewers and one score

FeatureSmith v0.4.0 runs 10 automated reviewers on Polars, covering schema, missingness, duplicates, cardinality, distributions, and more. Six of its detectors focus on leakage: target correlation, temporal leakage, and ID-shaped columns among them.

The findings roll up into an ML Readiness Score from 0 to 100 across 7 dimensions. The score is explainable: every point traces back to a finding, and the recommendations are ranked so you fix the costly problems first. A dataset diff shows what changed between two versions, and a diff-aware review checks only what moved.

Gating CI before the GPU hours

A review only helps if it runs every time. The CLI exits with codes 0 to 4 depending on what it finds and can write JSON, so a GitHub Actions job can stop a bad dataset before an expensive training job starts. The same engine is available as a Python SDK for notebooks and pipelines.

It ships on PyPI as featuresmith-cli and featuresmith-core, under Apache-2.0.

AI explains, it never decides

The tempting move was to let an LLM judge datasets. I did the opposite. Every finding is computed deterministically, so the same dataset gives the same result every run. An optional LLM only rewrites structured findings in plain language. The tool runs fully with AI off, and your data never leaves your machine.

For a tool that sits in CI and blocks merges, trust comes first. AI earns its place as the explainer, not the judge.