k-NN — k-Nearest Neighbours

Task	Classification
Method key (analysis_type)	knn
Prediction on new data	Yes — see Prediction
Library	sklearn.neighbors.KNeighborsClassifier, StandardScaler, Pipeline

When to use

• Samples of one class form compact groups in the colour space, of any shape.
• You want a simple method with an intuitive explanation: "this sample looks like these five known samples".
• Few features: in high dimensions all distances become similar and k-NN loses its power.

How it works

k-NN does not build a model. It stores the training samples. To classify a new sample $x$, it finds the $k$ training samples closest to $x$ and takes a vote of their classes. With uniform weights every neighbour has one vote; with distance weights a neighbour at distance $d$ has weight $1/d$, so the closest neighbours count more. The probability of class $c$ is the weighted share of neighbours of class $c$:

$$\hat{P}(c \mid x) = \frac{\sum_{i \in N_k(x)} w_i \,[y_i = c]}{\sum_{i \in N_k(x)} w_i} .$$

Distances between samples $x$ and $x'$ with $p$ features:

UI label	metric	Distance
Minkowski	minkowski (with $q = 2$)	$\left(\sum_j \lvert x_j - x'_j\rvert^{q}\right)^{1/q}$; with $q=2$ it equals the Euclidean distance
Euclidean	euclidean	$\sqrt{\sum_j (x_j - x'_j)^2}$
Manhattan	manhattan	$\sum_j \lvert x_j - x'_j\rvert$
Chebyshev	chebyshev	$\max_j \lvert x_j - x'_j\rvert$

Parameters

Defaults as set by the analysis dialog.

Parameter	UI label	Default	Notes
n_neighbors	Number of Neighbors	5	$k$. Small $k$: flexible boundary, sensitive to noise. Large $k$: smooth boundary, small classes get outvoted.
weights	Weight Function	uniform	uniform or distance
metric	Distance Metric	minkowski	See the table above
scale_data	Scale Data	on	Standardise features inside the model
algorithm	—	auto	Neighbour search algorithm; does not change the result
p	—	2	Power $q$ of the Minkowski distance
feature_selection	Enable Feature Selection	off	See Feature selection.
average_replicates	Average technical replicates	off	See Averaging replicates.

Preprocessing

With Scale Data on (default), the features are standardised (mean 0, standard deviation 1) by a StandardScaler inside a Pipeline. The scaler is fitted on the training rows of each fold only. Keep it on unless all features are already on the same scale: otherwise distances are dominated by the features with the largest range.

Before the model: replicate averaging and feature selection, if they are on. See Order of steps.

Results and metrics

The common classification metrics are described in Metrics. For binary problems, AUC uses the share of neighbours of the positive class as the score. With small $k$ this score takes only a few values (for $k = 5$: 0, 0.2, …, 1), so the ROC curve has few steps.

k-NN-specific fields:

Field	Meaning
prediction_probabilities	Weighted share of neighbours of each class, for every sample (final model)
model_info	The parameters actually used: n_neighbors, weights, algorithm, metric, p, scale_data

tip

With weights = distance, each training sample is its own nearest neighbour at distance 0, so the final model predicts its training data perfectly. The training metrics are then meaningless; look at the cross-validated ones.

Visualizations

• k-NN distance plot
• Confusion matrix and ROC curve
• Permutation plot
• Time series and heatmap of the input data

Source code

run_knn_analysis in chrometrica/analysis/analysis.py. The imports, the parameters, the model in the CV loop and the final model:

analysis.py · run_knn_analysis() · lines 4120–4356

def run_knn_analysis(X, y, parameters, cv_method, cv_folds):
    """Run K-Nearest Neighbors analysis with cross-validation and optional feature selection"""
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.pipeline import Pipeline
    from sklearn.preprocessing import StandardScaler
    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
    # ...
    n_neighbors = parameters.get('n_neighbors', 5)
    weights = parameters.get('weights', 'uniform')
    algorithm = parameters.get('algorithm', 'auto')
    metric = parameters.get('metric', 'minkowski')
    p = parameters.get('p', 2)
    scale_data = parameters.get('scale_data', True)
    # ...
    for train_idx, test_idx in cv.split(X_processed, y, groups=groups if cv_method == 'group' else None):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        # Create and train model
        if scale_data:
            knn_cv = Pipeline([
                ('scaler', StandardScaler()),
                ('knn', KNeighborsClassifier(
                    n_neighbors=n_neighbors,
                    weights=weights,
                    algorithm=algorithm,
                    metric=metric,
                    p=p
                ))
            ])
        else:
            knn_cv = KNeighborsClassifier(
                n_neighbors=n_neighbors,
                weights=weights,
                algorithm=algorithm,
                metric=metric,
                p=p
            )

        knn_cv.fit(X_train, y_train)
        y_pred = knn_cv.predict(X_test)
    # ...
    # Fit final model
    if scale_data:
        knn = Pipeline([
            ('scaler', StandardScaler()),
            ('knn', KNeighborsClassifier(
                n_neighbors=n_neighbors,
                weights=weights,
                algorithm=algorithm,
                metric=metric,
                p=p
            ))
        ])
    else:
        knn = KNeighborsClassifier(
            n_neighbors=n_neighbors,
            weights=weights,
            algorithm=algorithm,
            metric=metric,
            p=p
        )

    knn.fit(X_processed, y)
    y_pred_full = knn.predict(X_processed)

Full source of run_knn_analysis()

analysis.py · run_knn_analysis() · lines 4120–4497

def run_knn_analysis(X, y, parameters, cv_method, cv_folds):
    """Run K-Nearest Neighbors analysis with cross-validation and optional feature selection"""
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.pipeline import Pipeline
    from sklearn.preprocessing import StandardScaler
    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
    import numpy as np

    # ---------- Helper to sanitize JSON ----------
    def sanitize_for_json(obj):
        if isinstance(obj, dict):
            return {k: sanitize_for_json(v) for k, v in obj.items()}
        elif isinstance(obj, list):
            return [sanitize_for_json(v) for v in obj]
        elif isinstance(obj, float) and np.isnan(obj):
            return None
        elif isinstance(obj, (np.float32, np.float64)) and np.isnan(obj):
            return None
        elif isinstance(obj, np.ndarray):
            return sanitize_for_json(obj.tolist())
        else:
            return obj

    if parameters.get('average_replicates', False):
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("'groups' must be provided when averaging replicates is enabled.")
        X, y, new_groups = average_replicates(X, y, groups, method='mean')
        parameters['groups'] = new_groups

    # Get parameters with defaults
    n_neighbors = parameters.get('n_neighbors', 5)
    weights = parameters.get('weights', 'uniform')
    algorithm = parameters.get('algorithm', 'auto')
    metric = parameters.get('metric', 'minkowski')
    p = parameters.get('p', 2)
    scale_data = parameters.get('scale_data', True)

    # Get overall classes for consistent confusion matrix dimensions
    overall_classes = np.unique(y)
    overall_classes_list = overall_classes.tolist()
    n_overall_classes = len(overall_classes_list)
    is_binary = (n_overall_classes == 2)

    # Check for feature selection
    feature_selection_params = get_feature_selection_params(parameters)
    if feature_selection_params:
        X_processed, selected_features, feature_scores = select_features(
            X, y, feature_selection_params, 'classification'
        )
    else:
        X_processed = X
        selected_features = list(range(X.shape[1]))
        feature_scores = None

    # --- AUC initialisation (only used if binary) ---
    cv_auc_scores = []       # AUC per fold
    cv_scores_all = []       # decision scores / probabilities for all test samples

    # Enhanced cross-validation with metrics
    cv_scores = []
    cv_predictions = []
    cv_true_labels = []
    cv_class_reports = []
    cv_confusion_matrices = []
    cv_sensitivity_scores = []
    cv_specificity_scores = []

    if cv_method == 'stratified':
        cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=42)
    elif cv_method == 'group':
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("Groups parameter required for group cross-validation")
        cv = GroupKFold(n_splits=cv_folds)
    else:
        cv = KFold(n_splits=cv_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in cv.split(X_processed, y, groups=groups if cv_method == 'group' else None):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y[train_idx], y[test_idx]

        # Create and train model
        if scale_data:
            knn_cv = Pipeline([
                ('scaler', StandardScaler()),
                ('knn', KNeighborsClassifier(
                    n_neighbors=n_neighbors,
                    weights=weights,
                    algorithm=algorithm,
                    metric=metric,
                    p=p
                ))
            ])
        else:
            knn_cv = KNeighborsClassifier(
                n_neighbors=n_neighbors,
                weights=weights,
                algorithm=algorithm,
                metric=metric,
                p=p
            )

        knn_cv.fit(X_train, y_train)
        y_pred = knn_cv.predict(X_test)

        # ---------- AUC computation only for binary ----------
        if is_binary:
            # For binary we use predict_proba (positive class probability)
            if hasattr(knn_cv, "predict_proba"):
                y_prob = knn_cv.predict_proba(X_test)
                y_score = y_prob[:, 1]  # positive class probability
            elif hasattr(knn_cv, "decision_function"):
                y_score = knn_cv.decision_function(X_test)
            else:
                y_score = None

            if y_score is not None:
                try:
                    auc_fold = roc_auc_score(y_test, y_score)
                except ValueError:
                    auc_fold = None
                cv_auc_scores.append(auc_fold)
                if isinstance(y_score, np.ndarray):
                    cv_scores_all.extend(y_score.tolist())
                else:
                    cv_scores_all.extend(y_score)
            else:
                cv_auc_scores.append(None)
        else:
            # Multiclass: skip AUC entirely
            cv_auc_scores.append(None)

        # Store fold results
        cv_scores.append(accuracy_score(y_test, y_pred))
        cv_predictions.extend(y_pred)
        cv_true_labels.extend(y_test)

        fold_report = classification_report(y_test, y_pred, output_dict=True, zero_division=0)
        cv_class_reports.append(fold_report)

        cm = confusion_matrix(y_test, y_pred, labels=overall_classes_list)
        cv_confusion_matrices.append(cm)

        if is_binary:
            tn, fp, fn, tp = cm.ravel()
            sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
            specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
            cv_sensitivity_scores.append(sensitivity)
            cv_specificity_scores.append(specificity)
        else:
            cv_sensitivity_scores.append(None)
            cv_specificity_scores.append(None)

    # --- After CV loop: aggregated AUC only for binary ---
    auc_aggregated = None
    if is_binary and cv_scores_all:
        try:
            auc_aggregated = roc_auc_score(cv_true_labels, cv_scores_all)
        except ValueError:
            auc_aggregated = None
        if auc_aggregated is not None and np.isnan(auc_aggregated):
            auc_aggregated = None

    if is_binary:
        valid_auc = [a for a in cv_auc_scores if a is not None]
        cv_auc_mean = np.mean(valid_auc) if valid_auc else None
        cv_auc_std = np.std(valid_auc) if valid_auc else None
    else:
        cv_auc_mean = None
        cv_auc_std = None

    # ---- Continue with existing CV metrics (unchanged) ----
    cv_precision_scores = []
    cv_recall_scores = []
    cv_f1_scores = []

    for report in cv_class_reports:
        if 'macro avg' in report:
            cv_precision_scores.append(report['macro avg']['precision'])
            cv_recall_scores.append(report['macro avg']['recall'])
            cv_f1_scores.append(report['macro avg']['f1-score'])

    cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels, cv_predictions,
                                                      labels=overall_classes_list).tolist()

    if is_binary:
        cm_agg = np.array(cv_confusion_matrix_aggregated)
        tn, fp, fn, tp = cm_agg.ravel()
        cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
        cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        cv_sensitivity_aggregated = None
        cv_specificity_aggregated = None

    if cv_confusion_matrices:
        cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
            cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
        )
        if is_binary:
            cm_avg = np.array(cv_confusion_matrix_avg)
            tn, fp, fn, tp = cm_avg.ravel()
            cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
            cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
        else:
            cv_sensitivity_avg = None
            cv_specificity_avg = None
    else:
        cv_confusion_matrix_avg = []
        avg_matrix_classes = []
        cv_sensitivity_avg = None
        cv_specificity_avg = None

    # Fit final model
    if scale_data:
        knn = Pipeline([
            ('scaler', StandardScaler()),
            ('knn', KNeighborsClassifier(
                n_neighbors=n_neighbors,
                weights=weights,
                algorithm=algorithm,
                metric=metric,
                p=p
            ))
        ])
    else:
        knn = KNeighborsClassifier(
            n_neighbors=n_neighbors,
            weights=weights,
            algorithm=algorithm,
            metric=metric,
            p=p
        )

    knn.fit(X_processed, y)
    y_pred_full = knn.predict(X_processed)
    y_pred_proba = knn.predict_proba(X_processed) if hasattr(knn, 'predict_proba') else None

    if scale_data:
        knn_model = knn.named_steps['knn']
    else:
        knn_model = knn

    final_classes = knn_model.classes_.tolist()

    if cv_confusion_matrix_avg and avg_matrix_classes != final_classes:
        avg_to_final = {cls: idx for idx, cls in enumerate(avg_matrix_classes)}
        n_classes = len(final_classes)
        reordered_matrix = np.zeros((n_classes, n_classes))
        for i, true_cls in enumerate(final_classes):
            for j, pred_cls in enumerate(final_classes):
                if true_cls in avg_to_final and pred_cls in avg_to_final:
                    orig_i = avg_to_final[true_cls]
                    orig_j = avg_to_final[pred_cls]
                    if (orig_i < len(cv_confusion_matrix_avg) and
                            orig_j < len(cv_confusion_matrix_avg[0])):
                        reordered_matrix[i, j] = cv_confusion_matrix_avg[orig_i][orig_j]
        cv_confusion_matrix_avg = reordered_matrix.tolist()
        avg_matrix_classes = final_classes

    full_confusion_matrix = confusion_matrix(y, y_pred_full, labels=final_classes)
    if is_binary and full_confusion_matrix.shape == (2, 2):
        tn, fp, fn, tp = full_confusion_matrix.ravel()
        full_sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
        full_specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        full_sensitivity = None
        full_specificity = None

    validation_message = None
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        avg_sens_spec = (full_sensitivity + full_specificity) / 2
        accuracy = accuracy_score(y, y_pred_full)
        accuracy_diff = abs(accuracy - avg_sens_spec)
        if accuracy_diff > 0.01:
            validation_message = f"Note: Accuracy ({accuracy:.3f}) differs from (sensitivity + specificity)/2 ({avg_sens_spec:.3f}) by {accuracy_diff:.3f}"

    cv_sensitivity_scores_filtered = [s for s in cv_sensitivity_scores if s is not None]
    cv_sensitivity_mean = np.mean(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None
    cv_sensitivity_std = np.std(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None

    cv_specificity_scores_filtered = [s for s in cv_specificity_scores if s is not None]
    cv_specificity_mean = np.mean(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None
    cv_specificity_std = np.std(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None

    cv_confusion_matrices_lists = [cm.tolist() for cm in cv_confusion_matrices]

    # --- Full-data (resubstitution) AUC and accuracy, field parity with run_lda_analysis ---
    full_auc = None
    if is_binary and y_pred_proba is not None:
        try:
            pos_idx = final_classes.index(overall_classes_list[1])
            full_auc = roc_auc_score(y, y_pred_proba[:, pos_idx])
        except Exception:
            full_auc = None
    full_accuracy = accuracy_score(y, y_pred_full)

    results = {
        'method': 'K-Nearest Neighbors',
        'cv_scores': cv_scores,
        'cv_mean': np.mean(cv_scores) if cv_scores else 0.0,
        'cv_std': np.std(cv_scores) if cv_scores else 0.0,
        'cv_precision': np.mean(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_precision_std': np.std(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_recall': np.mean(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_recall_std': np.std(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_f1': np.mean(cv_f1_scores) if cv_f1_scores else 0.0,
        'cv_f1_std': np.std(cv_f1_scores) if cv_f1_scores else 0.0,
        # Sensitivity
        'cv_sensitivity': cv_sensitivity_mean,
        'cv_sensitivity_std': cv_sensitivity_std,
        'cv_sensitivity_aggregated': cv_sensitivity_aggregated,
        'cv_sensitivity_avg': cv_sensitivity_avg,
        'cv_sensitivity_scores': cv_sensitivity_scores,
        # Specificity
        'cv_specificity': cv_specificity_mean,
        'cv_specificity_std': cv_specificity_std,
        'cv_specificity_aggregated': cv_specificity_aggregated,
        'cv_specificity_avg': cv_specificity_avg,
        'cv_specificity_scores': cv_specificity_scores,
        # AUC (sanitized; only for binary)
        'cv_auc': sanitize_for_json(cv_auc_mean),
        'cv_auc_std': sanitize_for_json(cv_auc_std),
        'cv_auc_scores': sanitize_for_json(cv_auc_scores),
        'cv_auc_aggregated': sanitize_for_json(auc_aggregated),
        # Full-data AUC / accuracy (+ CV accuracy alias), field parity with LDA
        'auc': sanitize_for_json(full_auc),
        'accuracy': full_accuracy,
        'cv_accuracy': np.mean(cv_scores) if cv_scores else 0.0,
        # Validation
        'is_binary': is_binary,
        'validation_message': validation_message,
        # Full model metrics
        'sensitivity': full_sensitivity,
        'specificity': full_specificity,
        # Confusion matrices
        'cv_confusion_matrix': cv_confusion_matrix_avg,
        'cv_confusion_matrices': cv_confusion_matrices_lists,
        'cv_confusion_matrix_aggregated': cv_confusion_matrix_aggregated,
        'cv_class_reports': cv_class_reports,
        'classification_report': classification_report(y, y_pred_full, output_dict=True, zero_division=0),
        'confusion_matrix': full_confusion_matrix.tolist(),
        'labels': y.tolist(),
        'predictions': y_pred_full.tolist(),
        'prediction_probabilities': y_pred_proba.tolist() if y_pred_proba is not None else None,
        'classes': final_classes,
        'model_info': {
            'n_neighbors': n_neighbors,
            'weights': weights,
            'algorithm': algorithm,
            'metric': metric,
            'p': p,
            'scale_data': scale_data
        }
    }

    if feature_selection_params:
        results['feature_selection'] = {
            'selected_features': selected_features,
            'feature_scores': feature_scores,
            'method': feature_selection_params.get('method', 'anova'),
            'k': feature_selection_params.get('k', 'all')
        }

    if is_binary and full_sensitivity is not None and full_specificity is not None:
        results['metric_consistency_check'] = {
            'accuracy': accuracy_score(y, y_pred_full),
            'sensitivity': full_sensitivity,
            'specificity': full_specificity,
            'avg_sens_spec': (full_sensitivity + full_specificity) / 2,
            'difference': abs(accuracy_score(y, y_pred_full) - (full_sensitivity + full_specificity) / 2)
        }

    results = sanitize_for_json(results)

    # Final fitted model for inference on unknown samples (popped in run_analysis)
    return attach_model_bundle(results, knn, selected_features, feature_selection_params)

Limitations and common pitfalls

• Curse of dimensionality. With many features, the nearest and the farthest neighbours are almost equally far. Use feature selection or fewer channels.
• Imbalanced classes. A large class dominates the neighbourhood of most samples. Consider Balanced Random Forest instead.
• $k$ and classes. $k$ larger than the smallest class means that this class can never have a majority.
• Replicates. Without Group K-Fold, the nearest neighbour of a test well is usually its own replicate in the training fold, and CV accuracy is greatly inflated. See Replicates and Group K-Fold.

References

• Cover T., Hart P. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13, 21–27 (1967). doi:10.1109/TIT.1967.1053964
• scikit-learn user guide: Nearest neighbors classification.