When to use: Plan a train/validation/test split for a dataset, see the resulting sample count per class in each split, get a warning if any split is too small for reliable metrics, and get a recommended k-fold cross-validation setting based on your dataset size.
This tool plans a train/validation/test split for a dataset, converts your split percentages into actual sample counts, breaks those counts down per class so you can catch an under-sized split before you train anything, and recommends a k-fold cross-validation setting based on your dataset's total size.
The training set is what the model actually learns from. The test set is held out entirely until final evaluation, giving an unbiased estimate of how the model performs on unseen data — critically, it must never influence any decision made during development. The validation set sits between them: it's used repeatedly during development to tune hyperparameters, compare model architectures, or decide when to stop training, without touching the test set. Skipping the validation split and tuning directly against the test set is a common mistake that silently turns the test set into a second validation set — the reported test performance then overstates how the model will actually perform on truly new data, because you've implicitly fit choices to that data too.
A 70/15/15 split sounds balanced, but on a dataset with 500 total samples across 10 classes, the 15% test split works out to roughly 7-8 samples per class on average — far too few to reliably estimate per-class precision or recall, even though the percentage looks reasonable. This is why this planner reports per-class counts, not just percentages: the commonly-cited rule of thumb of at least ~30 samples per class per split is a rough guideline for getting a reasonably stable estimate of a metric, not a hard requirement, but splits well below that should be treated with real caution, especially for any per-class metric (not just overall accuracy).
A single fixed train/validation split "wastes" data — whatever ends up in validation never gets used for training, and the specific validation performance number can vary meaningfully just from which particular samples happened to land in that split, especially with smaller datasets. K-fold cross-validation addresses this by partitioning the data into k folds, training k separate times with each fold held out once as validation, and averaging the results — every sample gets used for both training and validation across the k runs, and the averaged metric is a more stable estimate than any single split. The trade-off is k times the training cost, which matters less for small/medium datasets (where the extra stability is valuable) and more for very large datasets or expensive-to-train models (where a single held-out split is usually stable enough on its own, and the extra compute isn't worth it).
Enter your total dataset size and the number of classes (use 1 for a regression problem, where "per class" doesn't apply the same way, or treat it as a rough sanity check on overall split size). Enter your desired train and test percentages — the validation percentage is whatever remains. Review the per-class counts and any warnings; if a split is under-sized, consider collecting more data for that class, reducing the number of classes (grouping rare classes), or switching to k-fold cross-validation instead of a fixed validation split, using the recommended k as a starting point.
70/15/15 or 80/10/10 are commonly used starting points, but the "right" split depends heavily on total dataset size — with abundant data (hundreds of thousands of samples or more), even a 98/1/1 split can leave plenty of samples in validation and test, while with limited data, you may need to lean more heavily toward k-fold cross-validation instead of "wasting" a large fixed chunk of your small dataset on validation alone. There is no universally correct percentage — check the resulting per-class counts (which this tool reports) rather than relying on percentage alone.
Yes, in almost all cases — a stratified split preserves the same class proportions in train, validation, and test as in the full dataset, which prevents the unlucky (and surprisingly common with random splitting on smaller datasets) scenario where a rare class ends up entirely in one split and missing from another. Most ML frameworks support this directly (e.g. scikit-learn's train_test_split has a stratify parameter) — this planner assumes a stratified split when computing per-class counts, since it divides total class count proportionally rather than assuming random variance.
Leave-one-out cross-validation (LOOCV) is the extreme case of k-fold where k equals the total sample count — every single sample gets its own turn as the sole validation point, training on all the rest. This gives the least-biased possible estimate of model performance for a given dataset, but at N training runs, it becomes computationally impractical once N is more than a few dozen to a couple hundred samples (depending on how expensive a single training run is) — which is why it's suggested only as a small-dataset option, not a general default.
No — k-fold CV is typically used in place of (or alongside) a validation split for model selection and hyperparameter tuning, but a separate, untouched test set is still standard practice for final, unbiased performance reporting. A common pattern is: hold out a test set first and set it aside completely, then run k-fold CV on the remaining data for all model development and tuning decisions, and only evaluate on the test set once, at the very end, after all decisions are finalized.
For a regression problem (predicting a continuous value rather than a class label), set classes to 1 — the tool will still report meaningful overall train/validation/test counts, just without a per-class breakdown that doesn't apply. For classification where class count is genuinely unknown at planning time, use your best current estimate; the tool is meant for planning before you commit to a specific split, and you can always recompute once your label set is finalized.
Try our AI and Data Science Studio
More calculators, simulators, and guides for this discipline.