Case study

FeatureSmith

Code review for datasets: catch leakage before you spend the GPU hours.

01 / Problem

The problem

Tabular datasets often carry target leakage, silent schema drift, missingness spikes, and distribution problems. Code goes through linters and review; datasets usually go straight into expensive training runs without either.

Building forecasting and reinforcement learning models, I kept losing time to dataset bugs: future timestamps leaking into features, ID columns correlated with the label.

02 / Solution

What I shipped

A Python SDK and CLI on Polars that runs 10 automated reviewers, scores datasets 0–100 for ML readiness across 7 dimensions, flags 6 leakage patterns, diffs dataset versions, and gates CI with exit codes 0–4.

10 automated dataset reviewers
0–100 ML Readiness Score across 7 dimensions
6 leakage detectors
Dataset diff and diff-aware review
Ranked recommendations traceable to findings, plus a `plan` primitive

03 / Architecture

How it’s built

SDK & CLI: Polars-backed loading and profiling with low memory overhead
Review engine: 10 automated reviewers for schema, missingness, duplicates, cardinality, distributions, and more
Leakage suite: 6 detectors for target correlation, temporal leakage, and ID-shaped leakage
ML Readiness Score: explainable 0–100 score across 7 dimensions, with ranked recommendations traceable to findings
Diff and CI gate: dataset diff, diff-aware review, and exit codes 0–4 with JSON output for pipelines

04 / Decisions

Key decisions

Kept every finding deterministic; AI only explains, so results are reproducible and the tool runs fully with AI off
Chose Polars over pandas for columnar speed and lower memory use
Positioned it as a developer tool, not a dashboard: CLI, SDK, and CI exit codes first
Strict MyPy, Ruff, and 300+ passing tests; PRD, ADRs, and governance live in the repo

05 / Outcome

By the numbers

automated reviewers: 10
leakage detectors: 6
passing tests: 300+

Shipped v0.4.0 on PyPI under Apache-2.0, with 300+ passing tests and strict MyPy.

06 / Roadmap

What’s next

v0.5
data contracts and a featuresmith.lock file
v0.6
dataset certification
v0.7
optional AI explanations (Ollama by default, or bring your own key)
v1.0
dbt, Feast, and MLflow exporters, plus a hosted tier