Data Science
A model-evaluation workbench that measures how classification pipelines behave under missingness, noise, distribution shift, and other real-world stresses.
View source repository
Why it matters
My role
Strong held-out accuracy does not reveal when a classifier stops working. FaultForge deliberately introduces missingness, numerical noise, distribution shift, categorical drift, unseen categories, prevalence change, and invariance probes.
Every severity level runs across multiple deterministic seeds and reports mean, standard deviation, minimum, and maximum. Conservative guards use mean minus standard deviation against user-defined requirements.
A failure becomes a breaking point only when all later severities also fail. Isolated drops are marked flaky, and the result is reported as an interval because a discrete sweep cannot prove an exact threshold.
The baseline includes accuracy, balanced accuracy, recall, precision, F1, ROC-AUC, PR-AUC, Brier score, and confusion matrices. Paired class-stratified bootstrap evidence prevents unsupported drops from being promoted as confirmed failures.
Calibration, probability margins, near-threshold risk, per-row prediction flips, curve-retention summaries, compound-stress ablations, and PSI feature-drift rankings help explain how degradation appears.
Deployment ranges prevent models from being penalized for irrelevant conditions. Fewer than 30 minority-class examples produces an inconclusive result, while fewer than 10 is rejected, keeping promotion evidence honest.
Immutable model and dataset fingerprints, full experiment configurations, deterministic seeds, version comparison rules, and standalone JSON/HTML evidence preserve the relationship between a decision and its exact inputs.
Technical implementation
Each layer connects an implementation choice to the decision or workflow it supports.
06 layers| Layer | Implementation | Operational purpose |
|---|---|---|
| Experiment definition | Versioned model artifacts, dataset fingerprints, target semantics, stress plans and metric guards | Make every evaluation reproducible and tied to the exact model and data used |
| Stress generation | Missingness, numerical noise, distribution shift, categorical drift, unseen categories and prevalence change | Measure behaviour under realistic data-quality and population changes |
| Repeated evaluation | Deterministic multi-seed trials with mean, standard deviation, minimum and maximum summaries | Separate stable degradation from one-off variation |
| Statistical evidence | Paired class-stratified bootstrap intervals and conservative guard values | Avoid declaring failure when the observed difference is not adequately supported |
| Diagnostics | Calibration, confusion matrices, probability margins, prediction flips, PSI and compound-stress ablations | Explain how and where model behaviour changes |
| Decision output | Pass, failure, flaky, inconclusive and execution-failure states with JSON and HTML evidence | Translate evaluation results into reviewable promotion decisions |
Product walkthrough
Documents the robustness confusion calibration stage of FaultForge and the evidence available to its user.
Documents the stress severity results stage of FaultForge and the evidence available to its user.
Organizes versioned model assets and keeps lineage and reliability evidence attached to each model.
Compares reference and challenger versions using evidence relevant to promotion decisions.
Turns a robustness evaluation into a guided, reproducible configuration workflow.
Documents the model detail stage of FaultForge and the evidence available to its user.
Documents the stress matrix stage of FaultForge and the evidence available to its user.
Documents the seaborn line stage of FaultForge and the evidence available to its user.