Logistic regression

Task	Classification
Method key (analysis_type)	logistic
Prediction on new data	Yes — see Prediction
Library	sklearn.linear_model.LogisticRegression, StandardScaler, Pipeline

When to use

• You need class probabilities, not only labels.
• The classes can be separated by a straight boundary.
• You want coefficients that show how each feature moves the prediction.

Logistic regression makes fewer assumptions than LDA: it does not require normally distributed features or equal spreads in the classes.

How it works

For two classes, the model estimates the probability of the positive class as

$$P(y = 1 \mid x) = \sigma(w^\top x + b) = \frac{1}{1 + e^{-(w^\top x + b)}} .$$

The weights $w$ and the intercept $b$ minimise the log-loss plus an L2 penalty:

$$\min_{w,b} \; \tfrac{1}{2}\lVert w \rVert^2 + C \sum_{i=1}^{n} \log\!\left(1 + e^{-y_i (w^\top x_i + b)}\right), \qquad y_i \in \{-1, 1\}.$$

For more than two classes, scikit-learn fits a multinomial (softmax) model: one weight vector per class, and the probabilities of all classes add up to 1. A sample is assigned to the class with the highest probability.

Parameters

Defaults as set by the analysis dialog. The model runs with random_state=42 and the scikit-learn defaults for everything else: L2 penalty, solver lbfgs.

Parameter	UI label	Default	Notes
C	Regularization (C)	1.0	Inverse strength of the L2 penalty. Smaller C: stronger penalty, smaller weights, smoother boundary; larger C: the model follows the training data more closely.
max_iter	Max Iterations	1000	Iteration limit of the solver. Raise it if the server log reports that the solver did not converge.
scale_data	—	true	Standardise features inside the model. Not shown in the dialog; always on.
feature_selection	Enable Feature Selection	off	See Feature selection.
average_replicates	Average technical replicates	off	See Averaging replicates.

Preprocessing

• Class labels are encoded as integers with LabelEncoder; results show the original labels.
• Features are standardised (mean 0, standard deviation 1) by a StandardScaler inside a Pipeline. The scaler is fitted on the training rows of each fold only, so the test fold does not leak into the scaling.
• Before the model: replicate averaging and feature selection, if they are on. See Order of steps.

Results and metrics

The common classification metrics are described in Metrics. For binary problems, AUC uses the predicted probability of the positive class.

Logistic-specific fields:

Field	Meaning
prediction_probabilities	Class probabilities for every sample (final model)
coefficients	Weights $w$ of the final model, one row per class (one row for binary). They refer to standardised features: compare their sizes directly, a larger absolute value means a stronger effect.
model_info	C, max_iter and scale_data actually used

Visualizations

• Confusion matrix and ROC curve
• Permutation plot
• Time series and heatmap of the input data

Source code

run_logistic_analysis in chrometrica/analysis/analysis.py. The imports, the model in the CV loop and the final model:

analysis.py · run_logistic_analysis() · lines 1400–1628

def run_logistic_analysis(X, y, parameters, cv_method, cv_folds):
    """Run Logistic Regression analysis with optional feature selection and scaling"""
    from sklearn.linear_model import LogisticRegression
    from sklearn.preprocessing import LabelEncoder, StandardScaler
    from sklearn.pipeline import Pipeline
    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
    import numpy as np
    # ...
    scale_data = parameters.get('scale_data', True)
    C = float(parameters.get('C', 1.0))
    max_iter = int(parameters.get('max_iter', 1000))

    # Encode labels for logistic regression
    le = LabelEncoder()
    y_encoded = le.fit_transform(y)
    # ...
    for train_idx, test_idx in cv.split(X_processed, y_encoded, groups=groups if cv_method == 'group' else None):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]

        # Create and train model with optional scaling
        if scale_data:
            lr_cv = Pipeline([
                ('scaler', StandardScaler()),
                ('logistic', LogisticRegression(random_state=42, max_iter=max_iter, C=C))
            ])
        else:
            lr_cv = LogisticRegression(random_state=42, max_iter=max_iter, C=C)

        lr_cv.fit(X_train, y_train)
        y_pred = lr_cv.predict(X_test)
    # ...
    # Fit final model on all data
    if scale_data:
        lr = Pipeline([
            ('scaler', StandardScaler()),
            ('logistic', LogisticRegression(random_state=42, max_iter=max_iter, C=C))
        ])
    else:
        lr = LogisticRegression(random_state=42, max_iter=max_iter, C=C)

    lr.fit(X_processed, y_encoded)
    y_pred_full = lr.predict(X_processed)
    y_pred_proba = lr.predict_proba(X_processed)

Full source of run_logistic_analysis()

analysis.py · run_logistic_analysis() · lines 1400–1762

def run_logistic_analysis(X, y, parameters, cv_method, cv_folds):
    """Run Logistic Regression analysis with optional feature selection and scaling"""
    from sklearn.linear_model import LogisticRegression
    from sklearn.preprocessing import LabelEncoder, StandardScaler
    from sklearn.pipeline import Pipeline
    from sklearn.model_selection import StratifiedKFold, GroupKFold, KFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score, roc_auc_score
    import numpy as np

    # ---------- Helper to sanitize JSON ----------
    def sanitize_for_json(obj):
        if isinstance(obj, dict):
            return {k: sanitize_for_json(v) for k, v in obj.items()}
        elif isinstance(obj, list):
            return [sanitize_for_json(v) for v in obj]
        elif isinstance(obj, float) and np.isnan(obj):
            return None
        elif isinstance(obj, (np.float32, np.float64)) and np.isnan(obj):
            return None
        elif isinstance(obj, np.ndarray):
            return sanitize_for_json(obj.tolist())
        else:
            return obj

    if parameters.get('average_replicates', False):
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("'groups' must be provided when averaging replicates is enabled.")
        X, y, new_groups = average_replicates(X, y, groups, method='mean')
        parameters['groups'] = new_groups

    # Get parameters with defaults
    scale_data = parameters.get('scale_data', True)
    C = float(parameters.get('C', 1.0))
    max_iter = int(parameters.get('max_iter', 1000))

    # Encode labels for logistic regression
    le = LabelEncoder()
    y_encoded = le.fit_transform(y)

    # Get overall classes for consistent confusion matrix dimensions
    overall_classes = np.unique(y)
    overall_classes_list = overall_classes.tolist()
    n_overall_classes = len(overall_classes_list)
    is_binary = (n_overall_classes == 2)

    # Check for feature selection
    feature_selection_params = get_feature_selection_params(parameters)
    if feature_selection_params:
        X_processed, selected_features, feature_scores = select_features(
            X, y_encoded, feature_selection_params, 'classification'
        )
    else:
        X_processed = X
        selected_features = list(range(X.shape[1]))
        feature_scores = None

    # --- AUC initialisation (only used if binary) ---
    cv_auc_scores = []       # AUC per fold
    cv_scores_all = []       # decision scores / probabilities for all test samples

    # Enhanced cross-validation with metrics
    cv_scores = []
    cv_predictions = []
    cv_true_labels = []
    cv_class_reports = []
    cv_confusion_matrices = []
    cv_sensitivity_scores = []
    cv_specificity_scores = []

    if cv_method == 'stratified':
        cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=42)
    elif cv_method == 'group':
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("Groups parameter required for group cross-validation")
        cv = GroupKFold(n_splits=cv_folds)
    else:
        cv = KFold(n_splits=cv_folds, shuffle=True, random_state=42)

    for train_idx, test_idx in cv.split(X_processed, y_encoded, groups=groups if cv_method == 'group' else None):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]

        # Create and train model with optional scaling
        if scale_data:
            lr_cv = Pipeline([
                ('scaler', StandardScaler()),
                ('logistic', LogisticRegression(random_state=42, max_iter=max_iter, C=C))
            ])
        else:
            lr_cv = LogisticRegression(random_state=42, max_iter=max_iter, C=C)

        lr_cv.fit(X_train, y_train)
        y_pred = lr_cv.predict(X_test)

        # ---------- AUC computation only for binary ----------
        if is_binary:
            # For binary we use predict_proba (positive class probability) or decision_function
            if hasattr(lr_cv, "predict_proba"):
                y_prob = lr_cv.predict_proba(X_test)
                y_score = y_prob[:, 1]  # positive class probability
            elif hasattr(lr_cv, "decision_function"):
                y_score = lr_cv.decision_function(X_test)
            else:
                y_score = None

            if y_score is not None:
                try:
                    auc_fold = roc_auc_score(y_test, y_score)
                except ValueError:
                    auc_fold = None
                cv_auc_scores.append(auc_fold)
                if isinstance(y_score, np.ndarray):
                    cv_scores_all.extend(y_score.tolist())
                else:
                    cv_scores_all.extend(y_score)
            else:
                cv_auc_scores.append(None)
        else:
            # Multiclass: skip AUC entirely
            cv_auc_scores.append(None)

        # Store fold results
        cv_scores.append(accuracy_score(y_test, y_pred))
        cv_predictions.extend(y_pred)
        cv_true_labels.extend(y_test)

        # Store per-fold classification report with original class labels
        fold_report = classification_report(y_test, y_pred, output_dict=True, zero_division=0)
        original_fold_report = {}
        for key, value in fold_report.items():
            if key.isdigit():
                class_idx = int(key)
                if class_idx < len(le.classes_):
                    original_key = le.classes_[class_idx]
                    original_fold_report[original_key] = value
            else:
                original_fold_report[key] = value
        cv_class_reports.append(original_fold_report)

        cm = confusion_matrix(y_test, y_pred, labels=range(n_overall_classes))
        cv_confusion_matrices.append(cm)

        if is_binary:
            tn, fp, fn, tp = cm.ravel()
            sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
            specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
            cv_sensitivity_scores.append(sensitivity)
            cv_specificity_scores.append(specificity)
        else:
            cv_sensitivity_scores.append(None)
            cv_specificity_scores.append(None)

    # --- After CV loop: compute aggregated AUC only for binary ---
    auc_aggregated = None
    if is_binary and cv_scores_all:
        try:
            auc_aggregated = roc_auc_score(cv_true_labels, cv_scores_all)
        except ValueError:
            auc_aggregated = None
        if auc_aggregated is not None and np.isnan(auc_aggregated):
            auc_aggregated = None

    # Compute mean/std of per-fold AUC (only binary)
    if is_binary:
        valid_auc = [a for a in cv_auc_scores if a is not None]
        cv_auc_mean = np.mean(valid_auc) if valid_auc else None
        cv_auc_std = np.std(valid_auc) if valid_auc else None
    else:
        cv_auc_mean = None
        cv_auc_std = None

    # ---- Continue with existing CV metrics (unchanged) ----
    cv_precision_scores = []
    cv_recall_scores = []
    cv_f1_scores = []

    for report in cv_class_reports:
        if 'macro avg' in report:
            cv_precision_scores.append(report['macro avg']['precision'])
            cv_recall_scores.append(report['macro avg']['recall'])
            cv_f1_scores.append(report['macro avg']['f1-score'])

    # Aggregated confusion matrix with original labels
    cv_true_labels_original = le.inverse_transform(cv_true_labels)
    cv_predictions_original = le.inverse_transform(cv_predictions)
    cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels_original, cv_predictions_original,
                                                      labels=overall_classes_list).tolist()

    if is_binary:
        cm_agg = np.array(cv_confusion_matrix_aggregated)
        tn, fp, fn, tp = cm_agg.ravel()
        cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
        cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        cv_sensitivity_aggregated = None
        cv_specificity_aggregated = None

    if cv_confusion_matrices:
        cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
            cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
        )
        if is_binary:
            cm_avg = np.array(cv_confusion_matrix_avg)
            tn, fp, fn, tp = cm_avg.ravel()
            cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
            cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
        else:
            cv_sensitivity_avg = None
            cv_specificity_avg = None
    else:
        cv_confusion_matrix_avg = []
        avg_matrix_classes = []
        cv_sensitivity_avg = None
        cv_specificity_avg = None

    # Fit final model on all data
    if scale_data:
        lr = Pipeline([
            ('scaler', StandardScaler()),
            ('logistic', LogisticRegression(random_state=42, max_iter=max_iter, C=C))
        ])
    else:
        lr = LogisticRegression(random_state=42, max_iter=max_iter, C=C)

    lr.fit(X_processed, y_encoded)
    y_pred_full = lr.predict(X_processed)
    y_pred_proba = lr.predict_proba(X_processed)

    if scale_data:
        lr_model = lr.named_steps['logistic']
    else:
        lr_model = lr

    final_classes = le.classes_.tolist()

    # Reorder average confusion matrix if needed
    if cv_confusion_matrix_avg and avg_matrix_classes != final_classes:
        avg_to_final = {cls: idx for idx, cls in enumerate(avg_matrix_classes)}
        n_classes = len(final_classes)
        reordered_matrix = np.zeros((n_classes, n_classes))
        for i, true_cls in enumerate(final_classes):
            for j, pred_cls in enumerate(final_classes):
                if true_cls in avg_to_final and pred_cls in avg_to_final:
                    orig_i = avg_to_final[true_cls]
                    orig_j = avg_to_final[pred_cls]
                    if (orig_i < len(cv_confusion_matrix_avg) and
                            orig_j < len(cv_confusion_matrix_avg[0])):
                        reordered_matrix[i, j] = cv_confusion_matrix_avg[orig_i][orig_j]
        cv_confusion_matrix_avg = reordered_matrix.tolist()
        avg_matrix_classes = final_classes

    y_pred_full_original = le.inverse_transform(y_pred_full)

    full_confusion_matrix = confusion_matrix(y, y_pred_full_original, labels=final_classes)
    if is_binary and full_confusion_matrix.shape == (2, 2):
        tn, fp, fn, tp = full_confusion_matrix.ravel()
        full_sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
        full_specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        full_sensitivity = None
        full_specificity = None

    validation_message = None
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        avg_sens_spec = (full_sensitivity + full_specificity) / 2
        accuracy = accuracy_score(y, y_pred_full_original)
        accuracy_diff = abs(accuracy - avg_sens_spec)
        if accuracy_diff > 0.01:
            validation_message = f"Note: Accuracy ({accuracy:.3f}) differs from (sensitivity + specificity)/2 ({avg_sens_spec:.3f}) by {accuracy_diff:.3f}"

    cv_sensitivity_scores_filtered = [s for s in cv_sensitivity_scores if s is not None]
    cv_sensitivity_mean = np.mean(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None
    cv_sensitivity_std = np.std(cv_sensitivity_scores_filtered) if cv_sensitivity_scores_filtered else None

    cv_specificity_scores_filtered = [s for s in cv_specificity_scores if s is not None]
    cv_specificity_mean = np.mean(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None
    cv_specificity_std = np.std(cv_specificity_scores_filtered) if cv_specificity_scores_filtered else None

    # --- Full-data (resubstitution) AUC and accuracy, by analogy with run_lda_analysis ---
    full_auc = None
    if is_binary:
        try:
            pos_idx = final_classes.index(overall_classes_list[1])
            full_auc = roc_auc_score(y, y_pred_proba[:, pos_idx])
        except Exception:
            full_auc = None
    full_accuracy = accuracy_score(y, y_pred_full_original)

    results = {
        'method': 'Logistic Regression',
        'cv_scores': cv_scores,
        'cv_mean': np.mean(cv_scores) if cv_scores else 0.0,
        'cv_std': np.std(cv_scores) if cv_scores else 0.0,
        'cv_precision': np.mean(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_precision_std': np.std(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_recall': np.mean(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_recall_std': np.std(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_f1': np.mean(cv_f1_scores) if cv_f1_scores else 0.0,
        'cv_f1_std': np.std(cv_f1_scores) if cv_f1_scores else 0.0,
        # Sensitivity
        'cv_sensitivity': cv_sensitivity_mean,
        'cv_sensitivity_std': cv_sensitivity_std,
        'cv_sensitivity_aggregated': cv_sensitivity_aggregated,
        'cv_sensitivity_avg': cv_sensitivity_avg,
        'cv_sensitivity_scores': cv_sensitivity_scores,
        # Specificity
        'cv_specificity': cv_specificity_mean,
        'cv_specificity_std': cv_specificity_std,
        'cv_specificity_aggregated': cv_specificity_aggregated,
        'cv_specificity_avg': cv_specificity_avg,
        'cv_specificity_scores': cv_specificity_scores,
        # AUC (sanitized; only for binary)
        'cv_auc': sanitize_for_json(cv_auc_mean),
        'cv_auc_std': sanitize_for_json(cv_auc_std),
        'cv_auc_scores': sanitize_for_json(cv_auc_scores),
        'cv_auc_aggregated': sanitize_for_json(auc_aggregated),
        # Full-data AUC / accuracy (+ CV accuracy alias), field parity with LDA
        'auc': sanitize_for_json(full_auc),
        'accuracy': full_accuracy,
        'cv_accuracy': np.mean(cv_scores) if cv_scores else 0.0,
        # Validation
        'is_binary': is_binary,
        'validation_message': validation_message,
        # Confusion matrices
        'cv_confusion_matrix': cv_confusion_matrix_avg,
        'cv_confusion_matrices': [cm.tolist() for cm in cv_confusion_matrices],
        'cv_confusion_matrix_aggregated': cv_confusion_matrix_aggregated,
        'cv_class_reports': cv_class_reports,
        'classification_report': classification_report(y, y_pred_full_original, output_dict=True, zero_division=0),
        'confusion_matrix': full_confusion_matrix.tolist(),
        'sensitivity': full_sensitivity,
        'specificity': full_specificity,
        'labels': y.tolist(),
        'predictions': y_pred_full_original.tolist(),
        'prediction_probabilities': y_pred_proba.tolist(),
        'classes': final_classes,
        'coefficients': lr_model.coef_.tolist() if hasattr(lr_model, 'coef_') else None,
        'model_info': {'scale_data': scale_data, 'C': C, 'max_iter': max_iter}
    }

    if feature_selection_params:
        results['feature_selection'] = {
            'selected_features': selected_features,
            'feature_scores': feature_scores,
            'method': feature_selection_params.get('method', 'anova'),
            'k': feature_selection_params.get('k', 'all')
        }

    if is_binary and full_sensitivity is not None and full_specificity is not None:
        results['metric_consistency_check'] = {
            'accuracy': accuracy_score(y, y_pred_full_original),
            'sensitivity': full_sensitivity,
            'specificity': full_specificity,
            'avg_sens_spec': (full_sensitivity + full_specificity) / 2,
            'difference': abs(accuracy_score(y, y_pred_full_original) - (full_sensitivity + full_specificity) / 2)
        }

    results = sanitize_for_json(results)

    # Final fitted model for inference on unknown samples (popped in run_analysis)
    return attach_model_bundle(results, lr, selected_features, feature_selection_params, label_encoder=le)

Limitations and common pitfalls

• Linear boundary. Like LDA, logistic regression separates classes with hyperplanes.
• Perfect separation. When the training classes are perfectly separable, the unpenalised weights would grow without limit. The L2 penalty keeps them finite, but the probabilities become very close to 0 and 1 and look more certain than they are.
• Correlated features. Colour channels are often strongly correlated. The predictions stay good, but the individual coefficients become unstable and hard to interpret.

References

• Cox D. R. The regression analysis of binary sequences. Journal of the Royal Statistical Society B, 20, 215–232 (1958). doi:10.1111/j.2517-6161.1958.tb00292.x
• Hastie T., Tibshirani R., Friedman J. The Elements of Statistical Learning, 2nd ed., section 4.4. Springer (2009).
• scikit-learn user guide: Logistic regression.