Cross-validation

Every supervised method (all methods except PCA) is evaluated with cross-validation (CV). The data are split into $k$ folds. The model is trained $k$ times, each time on $k-1$ folds, and tested on the remaining fold. Every sample is tested exactly once, by a model that did not see it during training.

You set the scheme in the Cross-Validation section of the analysis dialog: CV Method and Number of Folds.

Available schemes

UI label	cv_method	scikit-learn splitter	When to use
Group K-Fold	group	GroupKFold	Your data have technical replicates (several rows with the same sample_name). All rows of a sample go to the same fold. Default for a class label column.
Stratified K-Fold	stratified	StratifiedKFold	Classification without replicates. Every fold keeps the class proportions of the whole data set.
Random K-Fold	kfold	KFold	Rows are independent and classes are balanced.

StratifiedKFold and KFold shuffle the rows with a fixed seed (shuffle=True, random_state=42), so the same data always give the same folds. GroupKFold does not shuffle: it assigns whole groups to folds so that the folds have about the same number of rows.

The groups for Group K-Fold are always the values of the sample_name column, whatever label column you choose.

note

The database also knows the values none and leave_one_out, but the dialog does not offer them. If one of them is sent through the API, the backend treats it as Random K-Fold with the given number of folds.

Number of folds

When you choose the label column, the dialog sets the scheme and the number of folds for you:

Label column	CV Method	Number of Folds
A class column	Group K-Fold	Number of unique sample_name values in the selected rows
sample_name	Random K-Fold	Number of selected rows ÷ 6, rounded down

With Group K-Fold and one fold per sample, the default is leave-one-sample-out: each fold tests all replicates of one sample. This is the most honest estimate for small data sets, but it trains as many models as there are samples, so it is also the slowest.

You can change both values. Rules to keep in mind:

• Group K-Fold needs at least as many unique samples as folds.
• Stratified K-Fold needs at least as many rows in every class as folds. The dialog shows an error under Number of Folds and lists the classes that are too small.
• More folds mean larger training sets (a less pessimistic estimate) and more models to train (a slower analysis).

Group K-Fold and replicates

Replicates of one sample are much more similar to each other than to other samples. If one replicate is in the training set and another one in the test fold, the model "recognises" the sample instead of predicting its class, and the CV metrics look better than they will be on new samples. Group K-Fold prevents this. The background is in Replicates and Group K-Fold.

If you average replicates, every sample becomes one row and every row gets its own group. Group K-Fold then works like K-Fold without shuffling.

Order of steps

All classification methods follow the same order. The code is in chrometrica/analysis/analysis.py, one function run_<method>_analysis per method.

1. Prepare the data (run_analysis in tasks.py). Rows with an empty label are removed. Groups are built from sample_name.
2. Average replicates, if Average technical replicates is on.
3. Select features once, on all rows, if Enable Feature Selection is on. See Feature selection inside cross-validation.
4. Cross-validation loop. For every fold, a new model is created, fitted on the training rows and used to predict the test rows. Per-fold metrics and the predictions are collected.
5. Final model. One more model is fitted on all rows. Training metrics (accuracy, confusion_matrix, …) are computed on the same rows, and this is the model used for prediction on new samples.
6. Metrics. Cross-validated metrics are aggregated in three ways. See Metrics.

Source code

The splitter is chosen in the same way in every run_<method>_analysis function. Example from LDA:

analysis.py · run_lda_analysis() · lines 1042–1055

if cv_method == 'stratified':
    cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=42)
elif cv_method == 'group':
    groups = parameters.get('groups')
    if groups is None:
        raise ValueError("Groups parameter required for group cross-validation")
    cv = GroupKFold(n_splits=cv_folds)
else:
    from sklearn.model_selection import KFold
    cv = KFold(n_splits=cv_folds, shuffle=True, random_state=42)

for train_idx, test_idx in cv.split(X_processed, y, groups=groups if cv_method == 'group' else None):
    X_train, X_test = X_processed[train_idx], X_processed[test_idx]
    y_train, y_test = y[train_idx], y[test_idx]

The default scheme and number of folds are set in the analysis dialog (frontend/src/components/Experiment/AnalysisDialog.js, effect "Auto-update CV settings based on label column"):

AnalysisDialog.js · default CV settings

if (isClassColumn && stats && stats.uniqueCount > 0) {
  newCvMethod = 'group';
  newCvFolds = stats.uniqueCount;          // unique sample_name values
} else if (isSampleColumn && stats?.totalCount) {
  newCvMethod = 'kfold';
  newCvFolds = Math.floor(stats.totalCount / 6);
}

Choosing a scheme

1. Several rows per sample? Use Group K-Fold. Keep the default number of folds unless the analysis is too slow.
2. One row per sample, classes of different size? Use Stratified K-Fold with 5 or 10 folds.
3. One row per sample, balanced classes? Random K-Fold or Stratified K-Fold; the results are close.

Then check that the result is better than chance with a permutation test.