Directives de soumission d'échantillons
Why a High Validation Score Can Still Fail the Next Study
The main risk is often not the algorithm. It is what the algorithm was allowed to learn before evaluation.
Omics models are commonly built from far more molecular features than samples. Normalization, imputation, batch correction, feature filtering, differential analysis, dimensionality reduction, class balancing, and hyperparameter search can all learn from the data. If any learned step uses the complete dataset before splitting, the test set is no longer unseen.
Dependence can also enter through biology and study design. Repeated specimens from one donor, related individuals, matched tissue regions, technical replicates, plates, centers, or sequential time points may be closer to each other than to future samples. A random sample-level split can place near-duplicates on both sides of the evaluation boundary and answer an easier question than the intended research use.
We therefore audit the full route from raw or processed measurements to the final score. A model is not judged by one headline metric. We ask whether the evaluation unit matches the future use, whether every transformation was fitted only on training data, whether tuning was separated from assessment, and whether the result survives relevant shifts.
Signals that deserve an independent audit
- Feature selection occurred before cross-validation
- Only the best random split was reported
- Repeated samples were treated as independent
- Test data influenced normalization or batch correction
- No simple baseline was evaluated
- Only AUROC or correlation was reported
- The model weakened on a new cohort
- Code, model, and results cannot be reproduced together
Mangul et al. described systematic omics benchmarking as a comparison against suitable ground truth with standardized metrics, reusable execution, and explicit limitations. They also cautioned that simulation alone cannot reproduce all experimental variation. Read the Nature Communications benchmarking framework.
Six Decisions That Determine Whether the Model Is Trustworthy
Each audit question changes what the reported performance can legitimately support.
Can We Reproduce the Original Result?
Your decision: Is the reported score tied to a complete, executable pipeline or only to a figure and a partially documented analysis?
What we inspect: Data versions, sample identifiers, endpoint construction, feature definitions, preprocessing order, split manifests, seeds, software environment, hyperparameters, fitted objects, and prediction files.
What we do: We rerun the supplied pipeline where possible and compare sample counts, feature counts, fold assignments, predictions, and reported metrics. Differences are traced to code, data, environment, or undocumented manual choices.
Evidence basis: The systematic omics benchmarking framework recommends common data representation, parameter documentation, reusable execution, and transparent comparison conditions. Review the benchmarking principles.
Output: A reproducibility record that separates exact reproduction, approximate reconstruction, and elements that cannot be verified.
Did Test Information Enter the Model?
Your decision: Does the reported score measure generalization, or does it partly measure information reused from the held-out samples?
What we inspect: Global filtering, outcome-guided feature selection, normalization, imputation, batch correction, oversampling, dimensionality reduction, duplicate specimens, repeated donors, related samples, and post hoc split changes.
What we do: We draw a data-lineage map and label every operation as fixed, training-fitted, or test-dependent. Learned operations are rebuilt inside each training fold. Grouped splits keep linked samples together.
Evidence basis: Ambroise and McLachlan showed in gene-expression classification that cross-validation is biased when gene selection is performed outside the resampling loop. Correcting the selection process changed apparently negligible error into non-zero error. Read the PNAS study.
Output: A leakage register, corrected pipeline, and performance change attributable to each corrected boundary.
Was Model Selection Separated From Model Assessment?
Your decision: Was the same cross-validation result used both to choose the model and to claim its final performance?
What we inspect: Algorithm screening, feature-count selection, hyperparameter grids, early stopping, threshold selection, metric selection, seed selection, and repeated attempts on the same holdout.
What we do: When tuning is part of the model-building procedure, we place it in an inner loop and reserve outer folds for performance estimation. A final untouched cohort remains separate from model revision.
Evidence basis: Varma and Simon found that error estimates are biased when cross-validation is used to tune a classifier and the same result is reported as its error. Their nested procedure produced estimates close to an independent test set in their experiments. Read the BMC Bioinformatics study.
Output: A prespecified evaluation design, nested results, uncertainty across outer folds, and a locked final-model rule.
Does the Complex Model Beat a Relevant Baseline?
Your decision: Is the added complexity buying reliable information, or is a simpler rule performing similarly under the same test?
What we inspect: Majority or prevalence baselines, mean-value baselines, regularized linear models, single-layer models, biology-informed scores, and established task-specific methods.
What we do: Every candidate receives the same features available at prediction time, the same folds, tuning budget, and metrics. We report paired fold-level differences rather than comparing numbers from different splits.
Evidence basis: Cawley and Talbot showed that overfitting the model-selection criterion can produce performance differences comparable to those between learning algorithms. Fair assessment must therefore control the selection process, not only model training. Read the Journal of Machine Learning Research paper.
Output: A baseline ladder, paired performance comparison, complexity-versus-evidence assessment, and recommendation to retain or simplify the model.
Are Probabilities, Features, and Explanations Stable?
Your decision: Can the model support probability-based or feature-level research decisions, or only a limited ranking task?
What we inspect: Calibration curves, calibration intercept and slope, proper scoring rules, threshold sensitivity, bootstrap uncertainty, feature-selection frequency, rank stability, and explanation consistency across folds and cohorts.
What we do: We separate ranking from calibration. For feature-based conclusions, we repeat selection within resampling and quantify how often features, directions, and ranks recur. Explanations are interpreted as model behavior, not automatically as biological causation.
Evidence basis: Van Calster et al. argued that calibration is often neglected and must be assessed during validation, particularly when predictions are interpreted as probabilities. Read the BMC Medicine article. Nogueira et al. developed a statistical treatment of feature-selection stability, including uncertainty and comparison between selection procedures. Read the stability study.
Output: Calibration assessment, stability matrix, explanation-consistency report, and a clear statement of which outputs are sufficiently stable to interpret.
Where Does the Model Stop Generalizing?
Your decision: Does the model transfer to the cohort, site, platform, population, tissue, cell context, or perturbation that matters next?
What we inspect: Dataset provenance, participant overlap, assay and platform differences, missing features, label construction, prevalence, covariate shift, center effects, and unseen biological contexts.
What we do: We preserve external cohorts as untouched tests, report each cohort separately, and use leave-one-group-out or leave-one-site-out evaluation when that matches the intended use. If the model is revised after seeing an external cohort, that cohort becomes development evidence and a new untouched test is required.
Evidence basis: Whalen et al. described how dependence, sequencing depth, feature selection, and class balancing can bias genomics machine-learning evaluation. Read the Nature Reviews Genetics article. Rosenblatt et al. later quantified how several forms of leakage changed prediction performance across four datasets and three phenotypes. Read the Nature Communications study.
Output: Cohort-specific results, shift and failure analysis, observed use boundary, and the next validation setting required.
What We Benchmark and How We Isolate the Failure
A useful audit changes one source of optimism at a time, while keeping the research question fixed.
| Audit module | Questions tested | Methods selected by study design | Decision output |
|---|---|---|---|
| Provenance and reproducibility | Can the reported sample set, features, splits, predictions, and score be recreated? | Version inventory, identifier reconciliation, environment reconstruction, prediction-level checksums, metric recomputation | Reproduced, reconstructed with differences, or not reproducible |
| Leakage audit | Which transformations learned from test samples, and which biological units were split incorrectly? | Data-lineage tracing, fold-aware preprocessing, group split reconstruction, duplicate and overlap checks, ablation of suspect steps | Leakage map and corrected performance |
| Baseline benchmarking | Does the model improve on a simple or established comparator under identical conditions? | Prespecified naive, linear, biology-informed, and task-standard baselines; paired fold comparison | Evidence of added value or recommendation to simplify |
| Nested validation | Was tuning optimism separated from final assessment? | Nested cross-validation, repeated outer folds when justified, locked thresholds, untouched external evaluation | Less biased performance estimate with uncertainty |
| Metric and calibration review | Do the metrics match class balance, output type, and planned research decision? | Confusion matrices, precision-recall analysis, error metrics, rank metrics, calibration curves, proper scoring rules, threshold analysis | Metric set, calibrated-use statement, and threshold limits |
| Robustness and stability | Do predictions, selected features, and explanations persist under reasonable data variation? | Bootstrap or repeated resampling, seed sensitivity, perturbation tests, feature frequency, rank concordance, subgroup analysis | Stable findings, unstable components, and sensitivity drivers |
| Cross-cohort transfer | Where does performance change across site, platform, population, tissue, or experimental context? | Locked external tests, leave-one-group-out designs, cohort-specific calibration, shift characterization, missing-feature analysis | Transferability matrix and failure boundary |
| Documentation | Can another team understand what the model should and should not be used for? | Model card, data lineage, evaluation manifest, limitations, reproducible scripts, decision log | Handover-ready evidence package |
AI-assisted utilities can help trace pipeline dependencies, compare large prediction files, and surface unusual fold behavior. Our scientists determine whether a transformation is scientifically appropriate, whether a split matches the research unit, and whether a failure reflects implementation, measurement, or biology.
Benchmark Designs for Different Omics Prediction Tasks
The benchmark follows the decision the model is meant to support, not a fixed menu of algorithms.
Classification and molecular subtyping
We review class definitions, prevalence, repeated samples, threshold selection, and per-class errors. AUROC is not used alone when class imbalance or positive predictive value matters. For discovered subtypes, stability and external assignment rules are assessed separately from supervised prediction. Monti et al. introduced resampling-based consensus clustering to evaluate class number and membership stability in gene-expression data. Review the consensus clustering study.
Continuous response prediction
We compare mean or covariate-only baselines, absolute and squared error, correlation, residual patterns, and calibration of the prediction scale. A high correlation can coexist with biased magnitudes, so the claimed use determines which evidence matters. Van Calster et al. explain why discrimination alone does not establish calibrated predictions. Review the calibration evidence.
Time-to-event research models
We preserve time origin, censoring logic, and cohort eligibility. Evaluation can include discrimination over time, calibration at defined horizons, and group-aware resampling. Royston and Altman describe external validation of Cox models through both discrimination and calibration in an independent sample. Review the survival-model validation study. No model is presented for individual patient management.
Ranking and prioritization models
For targets, compounds, variants, or features, we test enrichment against a blinded reference set, rank stability, top-k sensitivity, and performance against simple evidence scores. Experimental confirmation is distinguished from agreement with a database used during development. Börnigen et al. showed that retrospective gene-prioritization benchmarks can be optimistic when underlying data sources already contain the association, and instead tested newly reported associations before database updates. Review the prospective-style prioritization benchmark.
Single-cell and perturbation models
Cell-level splits can be misleading when cells from the same donor or experiment appear in both sets. We define donor, batch, cell type, perturbation, dose, and species generalization separately and select metrics that reflect both expression magnitude and biological response. Wei et al. compared 27 methods across 29 datasets under cellular-context and perturbation-generalization scenarios. Review the single-cell perturbation benchmark.
Spatial and regulatory sequence models
We prevent neighboring regions, matched tissue sections, or related genomic windows from creating an easy test. Spatial continuity, domain agreement, robustness, cell-type context, and held-out chromosome, tissue, or donor designs are selected according to the intended claim. Yuan et al. benchmarked spatial clustering across diverse real and simulated spatial datasets. Review the spatial benchmark. DART-Eval evaluated regulatory sequence models on accessibility and functional tasks with appropriate supervised baselines. Review the regulatory benchmark.
When a New Experiment Is the Only Defensible Test
Existing data can expose an evaluation problem, but they cannot always create a truly unseen biological challenge.
We first search for suitable internal, partner, public, or reference data. A new experiment is proposed only when the model's next use requires a cohort, assay, perturbation, tissue, or ground truth that existing data do not contain. The experimental design is created around the locked model and the failure hypothesis. Samples generated for repair are kept separate from samples reserved for final testing.
| Evidence gap identified by the audit | Focused experimental option | Why the option is relevant |
|---|---|---|
| A coding-variant model lacks an independent sequence challenge set or suitable truth comparison. | Human and mouse whole exome sequencing | Zook and Salit described standardized variant benchmarking with reference data, matched representations, stratified performance, and explicit truth-set limits. The service is considered only for coding-variant questions and project-matched specimens. Review the variant benchmarking recommendations. |
| A regulatory sequence model needs measured chromatin-accessibility labels in a relevant cell context. | ATAC-seq | DART-Eval used accessibility measurements and biologically meaningful regulatory tasks to test whether complex DNA representations outperform appropriate supervised baselines. Review the DART-Eval benchmark. |
| A compound-response model was evaluated only on compounds or doses seen during development. | Drug-seq | Ye et al. demonstrated transcriptome-scale perturbation profiles across 433 compounds and eight doses, providing a direct basis for designing held-out compound, dose, or mechanism challenges. Review the DRUG-seq study. |
| A target-ranking or perturbation model lacks functional evidence for predicted regulators. | CRISPR screen sequencing | Recent single-cell perturbation benchmarks separate cellular-context and perturbation generalization and compare complex methods with multiple metrics and simple baselines. Review the perturbation-response benchmark. |
| A cell-state or response model has not been tested in new donors, cell types, or perturbation contexts. | Single-cell RNA sequencing | Wei et al. benchmarked generalizable perturbation prediction across datasets and out-of-distribution settings, showing why the held-out biological context must be defined explicitly. Read the study design and results. |
| A spatial model has only been tested on one tissue, section, platform, or simulated benchmark. | 10x spatial transcriptome sequencing | Yuan et al. evaluated 13 spatial clustering methods on 34 spatial datasets and additional challenging datasets using accuracy, continuity, marker detection, scalability, and robustness. Review the spatial benchmarking study. |
These are conditional evidence routes, not a standard package. Other measurements may be more appropriate for a specific model. We recommend an assay only after the audit states which claim it will test, what remains locked, how samples will be separated, and what result would change the client decision.
An Evidence-Preserving Workflow From Intake to Model Card
The workflow keeps reproduction, audit, repair, and independent testing visibly separate.
| Stage | What we do | Decision gate |
|---|---|---|
| 1. Define the claim | Lock the prediction target, unit of evaluation, intended research setting, available information at prediction time, and what constitutes success or failure. | Is the requested claim testable with the available evidence? |
| 2. Reproduce the model | Reconstruct data versions, environment, preprocessing, model fitting, thresholds, fold assignments, and reported results. | Can the original result be reproduced closely enough to audit? |
| 3. Audit evidence boundaries | Trace leakage, overlap, dependence, confounding, and model-selection reuse. Build corrected grouped or time-aware splits. | Does the original score remain interpretable? |
| 4. Run fair benchmarks | Evaluate simple and task-standard baselines, perform nested validation where needed, and report uncertainty using common splits. | Does complexity add repeatable value? |
| 5. Test behavior and transfer | Assess calibration, thresholds, feature stability, subgroup behavior, shift, and untouched cohort performance. | Where does the model work, weaken, or fail? |
| 6. Repair evidence if justified | Revise the pipeline or add a focused experimental benchmark. Repair data remain development evidence; final tests stay untouched. | Is another independent test required? |
| 7. Deliver the model card | Document intended use, data, evaluation protocol, results, limitations, use boundary, and next study. | Go, revise, restrict, collect evidence, or stop |
What to Provide for an Independent Audit
The strongest audit starts with prediction-level traceability, not only a saved model file.
| Input category | Preferred materials | Why it matters |
|---|---|---|
| Research question | Locked target, intended use, evaluation unit, planned population or experimental context, and decision threshold if used | Determines the correct split, metric, baseline, and external test |
| Data and metadata | Raw or processed matrices, sample identifiers, labels, donor or subject identifiers, replicate relationships, center, batch, platform, time, treatment, and exclusion records | Supports overlap, dependence, confounding, and shift audits |
| Pipeline | Code, notebooks, workflow files, configuration, environment or container, package versions, seeds, and execution instructions | Allows the result to be reconstructed and every learned step to be located |
| Model artifacts | Fitted model, feature list, transformations, selected thresholds, calibration object, and model version | Connects final predictions to the exact model under review |
| Evaluation records | Train, validation, and test manifests; fold assignments; tuning history; prediction files; metric code; prior benchmark figures | Reveals model-selection reuse and enables score recomputation |
| External evidence | Untouched internal or partner cohort, eligible public datasets, reference materials, or stored biospecimens | Determines whether transferability can be tested without generating new data |
| Access constraints | Data-use conditions, secure-compute requirements, permitted outputs, and code or model restrictions | Defines what can be inspected and the resulting evidence level |
Partial audits are possible when some artifacts are unavailable. We state which conclusions remain supportable and which risks cannot be ruled out. Biospecimen quantities and assay requirements are specified only after the model's evidence gap and experimental design are agreed; they are not inferred from the page.
Deliverables for a Go, Revise, Restrict, or Stop Decision
The report is built for action by data scientists, biologists, and program leads.
Evidence audit
- Data and model provenance
- Reproducibility findings
- Leakage and dependence map
- Split and metric justification
- Unverifiable components
Benchmark results
- Simple and task-standard baselines
- Nested or grouped validation
- Uncertainty and sensitivity
- Calibration and stability
- Cohort and subgroup results
Decision package
- Model card
- Observed use boundary
- Failure analysis
- Repair options
- New experiment or cohort plan
We distinguish five outcomes: retain the model for the tested research use; revise the pipeline and retest; restrict use to supported cohorts or outputs; generate a new benchmark; or stop using the model for the proposed claim. A negative result is retained when it is the most defensible conclusion.
Model cards were proposed to document intended uses, evaluation conditions, performance characteristics, and limitations. We adapt that idea to omics research by adding assay context, cohort provenance, feature availability, experimental dependencies, and the observed biological transfer boundary. Read the model-card framework.
References
- Mangul S, Martin LS, Hill BL, et al. Systematic benchmarking of omics computational tools. Nature Communications. 2019.
- Ambroise C, McLachlan GJ. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences. 2002.
- Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. 2006.
- Cawley GC, Talbot NLC. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research. 2010.
- Whalen S, Schreiber J, Noble WS, Pollard KS. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics. 2022.
- Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019.
- Nogueira S, Sechidis K, Brown G. On the stability of feature selection algorithms. Journal of Machine Learning Research. 2018.
- Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D. Data leakage inflates prediction performance in connectome-based machine learning models. Nature Communications. 2024.
- Mitchell M, Wu S, Zaldivar A, et al. Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019.
- Zook JM, Salit M. Best practices for benchmarking germline small-variant calls in human genomes. Nature Biotechnology. 2019.
- Ye C, Ho DJ, Neri M, et al. DRUG-seq for miniaturized high-throughput transcriptome profiling in drug discovery. Nature Communications. 2018.
- Wei Z, Wang Y, Gao Y, et al. Benchmarking algorithms for generalizable single-cell perturbation response prediction. Nature Methods. 2026.
- Yuan Z, Zhao F, Lin S, et al. Benchmarking spatial clustering methods with spatially resolved transcriptomics data. Nature Methods. 2024.
- Monti S, Tamayo P, Mesirov J, Golub T. Consensus clustering: a resampling-based method for class discovery and visualization of gene expression microarray data. Machine Learning. 2003.
- Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Medical Research Methodology. 2013.
- Börnigen D, Tranchevent LC, Bonachela-Capdevila F, et al. An unbiased evaluation of gene prioritization tools. Bioinformatics. 2012.
Example Omics Model Audit Report
The demo report compares the published or internal claim with a reproducible benchmark under corrected evidence boundaries. Results remain separated by fold, cohort, and research context.
A project-specific report may include an evidence-boundary diagram, reproduction delta, reported-versus-corrected performance, baseline ladder, nested cross-validation distribution, calibration curve, threshold table, feature-selection stability, subgroup results, cross-cohort transfer matrix, and model card. The aim is to show which part of the evidence changed and why, not simply to replace one score with another.
Omics AI Model Benchmarking and Validation FAQs
1. Can you audit a model if we only have predictions and no source code?
We can review prediction files, split manifests, metrics, calibration, cohort behavior, and available documentation. We cannot rule out code-level leakage or reproduce hidden preprocessing without the relevant artifacts, so the final report states that limitation.
2. Do you rebuild the model during validation?
We first reproduce and audit the locked model. If leakage or an implementation problem is found, remediation is treated as a separate development step. A repaired model is evaluated again and requires an untouched test if independent validation is claimed.
3. Is nested cross-validation always required?
No. It is useful when feature selection, hyperparameters, thresholds, or algorithm choices are tuned using resampling and an unbiased internal performance estimate is needed. A fixed model evaluated once in a genuinely untouched external cohort does not need another tuning loop.
4. Which performance metric will you use?
The metric follows the output and decision. Classification, regression, ranking, time-to-event, perturbation, and spatial tasks require different primary metrics and baselines. We usually report several complementary measures and avoid selecting a metric after seeing which one looks best.
5. Can a model have good discrimination but poor calibration?
Yes. A model may rank samples in a useful order while assigning probabilities that are too high or too low. If probabilities or thresholds support the research decision, calibration must be assessed in the relevant setting.
6. How do you evaluate feature importance?
We repeat feature selection and explanation within resampling, quantify selection and rank stability, and compare directions across folds and cohorts. Importance describes how the fitted model behaves; it does not by itself establish that a feature causes the biological outcome.
7. What if the model fails in an external cohort?
We retain the result and test plausible causes such as implementation differences, missing features, platform shift, label construction, prevalence, center effects, sample composition, or a genuinely narrower biological use. We do not tune on the failed cohort and still describe it as independent validation.
8. Can you work with public data?
Yes, when provenance, participant overlap, feature coverage, labels, and use conditions support the planned test. A public dataset used during model development is not an untouched external test.
9. When do you recommend new sequencing or another omics assay?
Only when the audit identifies a specific missing context or ground truth that existing data cannot provide. The new study is designed around a locked claim, defined sample unit, prespecified analysis, and clear result that would change the decision.
10. What is included in the model card?
It documents intended research use, data and assay context, model version, inputs, evaluation protocol, baselines, metrics, calibration, stability, cohort-specific results, known limitations, unsupported uses, and the next evidence required.
Published Case Study
Independent Research Highlight
How Feature Leakage Changed an Apparently Predictive Model
This publication is an independent research example. It is not a CD Genomics customer project.
Background
Rosenblatt et al. investigated how data leakage changes connectome-based machine-learning results. The study covered four public datasets, three phenotypes, and more than 400 analysis pipelines. The tested problems included feature selection, covariate-related preprocessing, family structure, and repeated subjects.
Methods
The authors used ridge regression with five-fold cross-validation, selection of 5% of features, and a grid search for the regularization parameter. Their leakage-controlled reference pipeline performed covariate regression, site correction, and feature selection inside cross-validation, while split construction accounted for family structure. Figure 3 compared feature selection performed correctly inside the folds with feature selection performed on the combined training and test data.
Results
For attention-problem prediction in the HCPD dataset, the leakage-controlled pipeline was at chance level with median r = 0.01 and q2 = −0.13. Leaky feature selection changed the reported result to r = 0.48 and q2 = 0.22, corresponding to Δr = 0.47 and Δq2 = 0.35. The study also found that 20% repeated-subject leakage inflated performance across datasets and phenotypes, although the size of the change depended on the task.
Why It Matters
The algorithm, phenotype, and underlying data can remain the same while one misplaced feature-selection step changes the apparent conclusion. The study also showed that leakage does not always inflate every metric; some covariate-related leakage reduced performance. An audit must therefore reconstruct each boundary and measure its effect rather than assume one universal correction.
Conclusion
Figure 3 provides a concrete example of why feature selection belongs inside the resampling loop. A corrected score is not a penalty applied to the original result. It is a different and more defensible estimate of performance on unseen data.
Open-access note: The article is licensed under the Creative Commons Attribution 4.0 International License. This page describes the verified study and Figure 3 without reproducing the figure.
Reference
- Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D. Data leakage inflates prediction performance in connectome-based machine learning models. Nature Communications. 2024.
Selected Publications
These independent publications provide the methodological and experimental basis for this Solution. They are not presented as CD Genomics customer projects.
- Mangul S, Martin LS, Hill BL, et al. Systematic benchmarking of omics computational tools. Nature Communications. 2019.
- Ambroise C, McLachlan GJ. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the National Academy of Sciences. 2002.
- Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. 2006.
- Cawley GC, Talbot NLC. On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research. 2010.
- Whalen S, Schreiber J, Noble WS, Pollard KS. Navigating the pitfalls of applying machine learning in genomics. Nature Reviews Genetics. 2022.
- Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019.
- Nogueira S, Sechidis K, Brown G. On the stability of feature selection algorithms. Journal of Machine Learning Research. 2018.
- Rosenblatt M, Tejavibulya L, Jiang R, Noble S, Scheinost D. Data leakage inflates prediction performance in connectome-based machine learning models. Nature Communications. 2024.
- Mitchell M, Wu S, Zaldivar A, et al. Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency. 2019.
- Zook JM, Salit M. Best practices for benchmarking germline small-variant calls in human genomes. Nature Biotechnology. 2019.
- Ye C, Ho DJ, Neri M, et al. DRUG-seq for miniaturized high-throughput transcriptome profiling in drug discovery. Nature Communications. 2018.
- Wei Z, Wang Y, Gao Y, et al. Benchmarking algorithms for generalizable single-cell perturbation response prediction. Nature Methods. 2026.
- Yuan Z, Zhao F, Lin S, et al. Benchmarking spatial clustering methods with spatially resolved transcriptomics data. Nature Methods. 2024.
- Monti S, Tamayo P, Mesirov J, Golub T. Consensus clustering: a resampling-based method for class discovery and visualization of gene expression microarray data. Machine Learning. 2003.
- Royston P, Altman DG. External validation of a Cox prognostic model: principles and methods. BMC Medical Research Methodology. 2013.
- Börnigen D, Tranchevent LC, Bonachela-Capdevila F, et al. An unbiased evaluation of gene prioritization tools. Bioinformatics. 2012.
