XGBoost — Extreme Gradient Boosting

Task	Classification
Method key (analysis_type)	xgboost
Prediction on new data	Yes — see Prediction
Library	xgboost.XGBClassifier (parameters)

When to use

• Larger data sets (hundreds of samples and more) with non-linear relations.
• You are ready to tune hyper-parameters: XGBoost has many, and the defaults are rarely the best choice for small chemometric data sets.

For small data sets, random forest or PLS-DA are usually more robust.

How it works

Gradient boosting builds decision trees one after another. Each new tree $f_m$ is fitted to correct the errors of the trees before it, and its prediction is added with a small weight, the learning rate $\eta$:

$$F_m(x) = F_{m-1}(x) + \eta\, f_m(x) .$$

XGBoost chooses each tree by minimising a regularised objective: the log-loss of the predictions plus a penalty on the tree complexity

$$\Omega(f) = \gamma\, T + \tfrac{1}{2} \lambda \lVert w \rVert^2 + \alpha \lVert w \rVert_1 ,$$

where $T$ is the number of leaves and $w$ the leaf values. For two classes the model outputs a probability through the logistic function; for more classes, one set of trees per class and a softmax.

Early stopping. If the score on an evaluation set has not improved for early_stopping_rounds trees, training stops and the best number of trees is kept.

Parameters

Defaults as set by the analysis dialog.

Parameter	UI label	Default	Notes
n_estimators	Number of Trees	100	Maximum number of trees
learning_rate	Learning Rate	0.1	$\eta$
max_depth	Max Depth	6	Depth of each tree
min_child_weight	Min Child Weight	1	Minimum sum of instance weights in a leaf
subsample	Subsample	1.0	Share of rows used for each tree
colsample_bytree	Column Sample	1.0	Share of features used for each tree
gamma	Gamma	0	$\gamma$: minimum loss reduction for a split
reg_alpha	Alpha (L1)	0	$\alpha$
reg_lambda	Lambda (L2)	1	$\lambda$
early_stopping_rounds	Early Stopping Rounds	10	0 turns early stopping off. See Early stopping and validation.
eval_metric	Evaluation Metric	mlogloss	Metric for early stopping. Converted automatically: for two classes mlogloss → logloss, merror → error.
objective	Objective Function	multi:softprob	multi:softprob and multi:softmax also work for two classes. Binary Classification (binary:logistic) with more than two classes is replaced by multi:softprob.
booster	Booster	gbtree	gbtree: gradient boosted trees; gblinear: linear models instead of trees; dart: trees with dropout
tree_method	Tree Method	auto	Algorithm for building the trees: auto, exact, approx, hist. Does not apply to gblinear.
random_state	—	42
feature_selection	Enable Feature Selection	off	See Feature selection.
average_replicates	Average technical replicates	off	See Averaging replicates.

The model always runs with n_jobs=1.

Preprocessing

• Class labels are encoded as integers 0, 1, … with LabelEncoder; results show the original labels.
• No scaling: trees do not need it.
• Before the model: replicate averaging and feature selection, if they are on. See Order of steps.

Early stopping and validation

With early_stopping_rounds > 0 (the default is 10), every model, in each CV fold and the final one, holds out a random 10 % of its own training rows as the early-stopping evaluation set and is trained on the other 90 %. The test fold is never used to choose the number of trees, so the CV metrics stay honest.

• The validation part is stratified by class when every class has at least two rows.
• With Group K-Fold it is drawn by sample (all replicates of a sample are on the same side), so replicates do not leak into the evaluation set.
• If no usable validation part can be made (too few rows, or a class would disappear from the training part), the model is trained on all its rows without early stopping.

With few rows, a 10 % validation part is only a handful of samples and early stopping becomes noisy. Setting Early Stopping Rounds to 0 then trains all Number of Trees trees on all rows.

Results and metrics

The common classification metrics are described in Metrics. AUC is not computed for XGBoost.

XGBoost-specific fields:

Field	Meaning
feature_importances	Importance of each feature in the final model (feature_importances_, XGBoost default importance type)
cv_feature_importances	The same importances, averaged over the CV fold models
prediction_probabilities	Class probabilities for every sample (final model)
xgboost_parameters	All parameters actually passed to XGBClassifier
best_iteration, best_score	Best number of trees and its evaluation score, if early stopping was used

Visualizations

The visualization dialog offers no plots for XGBoost in the current version.

Source code

run_xgboost_analysis in chrometrica/analysis/analysis.py. The imports, the parameters, the model in the CV loop and the final model:

analysis.py · run_xgboost_analysis() · lines 6365–6594

def run_xgboost_analysis(X, y, parameters, cv_method, cv_folds):
    """Run XGBoost analysis with cross-validation and optional feature selection"""
    import xgboost as xgb
    from sklearn.model_selection import StratifiedKFold, GroupKFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
    from sklearn.preprocessing import LabelEncoder
    # ...
    xgb_params = {
        'objective': parameters.get('objective', 'multi:softprob'),
        'learning_rate': parameters.get('learning_rate', 0.1),
        'max_depth': parameters.get('max_depth', 6),
        'min_child_weight': parameters.get('min_child_weight', 1),
        'subsample': parameters.get('subsample', 1.0),
        'colsample_bytree': parameters.get('colsample_bytree', 1.0),
        'gamma': parameters.get('gamma', 0),
        'reg_alpha': parameters.get('reg_alpha', 0),
        'reg_lambda': parameters.get('reg_lambda', 1),
        'n_estimators': parameters.get('n_estimators', 100),
        'random_state': parameters.get('random_state', 42),
        'n_jobs': 1,  # Use 1 CPU core to avoid multiprocessing issues with Celery
        'eval_metric': parameters.get('eval_metric', 'mlogloss'),
        'num_class': len(class_names),
        'use_label_encoder': False,  # Disable label encoder warning
        'early_stopping_rounds': parameters.get('early_stopping_rounds', 10),
        'tree_method': parameters.get('tree_method', 'auto'),
        'booster': parameters.get('booster', 'gbtree')
    }

    # Make objective, num_class and eval_metric consistent with the number of classes
    # (xgboost fails on 'binary:*' objective with num_class, and on binary metrics
    # 'error'/'logloss' with 'multi:*' objectives and vice versa)
    if len(class_names) == 2:
        xgb_params['objective'] = parameters.get('objective', 'binary:logistic')

    if len(class_names) > 2 and str(xgb_params['objective']).startswith('binary:'):
        logger.warning(f"XGBoost: objective '{xgb_params['objective']}' is binary, but there are "
                       f"{len(class_names)} classes - using 'multi:softprob'")
        xgb_params['objective'] = 'multi:softprob'

    if str(xgb_params['objective']).startswith('binary:'):
        # Binary objective: no num_class, binary metrics
        xgb_params.pop('num_class', None)
        metric_map = {'mlogloss': 'logloss', 'merror': 'error'}
    else:
        # Multi-class objective (also valid for 2 classes): multi-class metrics
        metric_map = {'logloss': 'mlogloss', 'error': 'merror'}
    xgb_params['eval_metric'] = metric_map.get(xgb_params['eval_metric'], xgb_params['eval_metric'])
    # ...
    for fold_idx, (train_idx, test_idx) in enumerate(cv.split(X_processed, y_encoded, groups=groups_data)):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]
        y_test_original = y[test_idx]

        # Train and predict. Early stopping uses a validation part of the
        # training rows only: the test fold must not choose the number of trees.
        xgb_cv = xgb.XGBClassifier(**xgb_params)
        _fit_xgb_with_early_stopping(
            xgb_cv, X_train, y_train,
            early_stopping_rounds=xgb_params.get('early_stopping_rounds', 10),
            random_state=xgb_params['random_state'],
            groups=groups_data[train_idx] if groups_data is not None else None
        )
    # ...
    # Fit final model on all data
    xgb_final = xgb.XGBClassifier(**xgb_params)

    # Early stopping on a 10 % validation part (grouped by sample for Group
    # K-Fold); without early stopping, or if no validation part can be made,
    # the model is fitted on all rows.
    _fit_xgb_with_early_stopping(
        xgb_final, X_processed, y_encoded,
        early_stopping_rounds=xgb_params.get('early_stopping_rounds', 10),
        random_state=xgb_params['random_state'],
        groups=groups_data
    )

Early stopping on a validation part of the training rows:

analysis.py · _fit_xgb_with_early_stopping() · lines 6325–6362

def _fit_xgb_with_early_stopping(model, X, y, early_stopping_rounds, random_state,
                                 groups=None, val_size=0.1):
    """
    Fit an XGBClassifier with early stopping on a validation part split off
    (X, y) - never on the rows the model is evaluated on. With groups
    (Group K-Fold) the split keeps all rows of a group together.

    Without early stopping (early_stopping_rounds <= 0), or when no usable
    validation part can be made (too few rows, a class would disappear from
    the fitting part), the model is fitted on all of (X, y) without it.
    """
    if early_stopping_rounds and early_stopping_rounds > 0:
        from sklearn.model_selection import train_test_split, GroupShuffleSplit
        all_classes = np.unique(y)
        fit_idx = val_idx = None
        try:
            if groups is not None:
                splitter = GroupShuffleSplit(n_splits=1, test_size=val_size, random_state=random_state)
                fit_idx, val_idx = next(splitter.split(X, y, groups))
            else:
                _, counts = np.unique(y, return_counts=True)
                fit_idx, val_idx = train_test_split(
                    np.arange(len(y)), test_size=val_size,
                    stratify=y if counts.min() >= 2 else None,
                    random_state=random_state
                )
        except ValueError:
            fit_idx = val_idx = None

        # XGBoost expects every class label 0..K-1 in the fitting part
        if (fit_idx is not None and len(val_idx) > 0
                and np.array_equal(np.unique(y[fit_idx]), all_classes)):
            model.fit(X[fit_idx], y[fit_idx], eval_set=[(X[val_idx], y[val_idx])], verbose=False)
            return model

    model.set_params(early_stopping_rounds=None)
    model.fit(X, y)
    return model

Full source of run_xgboost_analysis()

analysis.py · run_xgboost_analysis() · lines 6365–6756

def run_xgboost_analysis(X, y, parameters, cv_method, cv_folds):
    """Run XGBoost analysis with cross-validation and optional feature selection"""
    import xgboost as xgb
    from sklearn.model_selection import StratifiedKFold, GroupKFold
    from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
    from sklearn.preprocessing import LabelEncoder
    import numpy as np

    if parameters.get('average_replicates', False):
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("'groups' must be provided when averaging replicates is enabled.")
        X, y, new_groups = average_replicates(X, y, groups, method='mean')
        # Update parameters with new groups (now each row is unique)
        parameters['groups'] = new_groups
        # If you used group CV, you might want to keep it; but with unique groups,
        # GroupKFold becomes equivalent to standard KFold. You can leave it as is.


    # Get overall classes for consistent confusion matrix dimensions
    overall_classes = np.unique(y)
    overall_classes_list = overall_classes.tolist()
    n_overall_classes = len(overall_classes_list)

    # Check for feature selection
    feature_selection_params = get_feature_selection_params(parameters)
    if feature_selection_params:
        X_processed, selected_features, feature_scores = select_features(
            X, y, feature_selection_params, 'classification'
        )
    else:
        X_processed = X
        selected_features = list(range(X.shape[1]))
        feature_scores = None

    # Encode string labels to integers for XGBoost
    le = LabelEncoder()
    y_encoded = le.fit_transform(y)
    class_names = le.classes_.tolist()

    # Get XGBoost parameters from analysis parameters with sensible defaults
    xgb_params = {
        'objective': parameters.get('objective', 'multi:softprob'),
        'learning_rate': parameters.get('learning_rate', 0.1),
        'max_depth': parameters.get('max_depth', 6),
        'min_child_weight': parameters.get('min_child_weight', 1),
        'subsample': parameters.get('subsample', 1.0),
        'colsample_bytree': parameters.get('colsample_bytree', 1.0),
        'gamma': parameters.get('gamma', 0),
        'reg_alpha': parameters.get('reg_alpha', 0),
        'reg_lambda': parameters.get('reg_lambda', 1),
        'n_estimators': parameters.get('n_estimators', 100),
        'random_state': parameters.get('random_state', 42),
        'n_jobs': 1,  # Use 1 CPU core to avoid multiprocessing issues with Celery
        'eval_metric': parameters.get('eval_metric', 'mlogloss'),
        'num_class': len(class_names),
        'use_label_encoder': False,  # Disable label encoder warning
        'early_stopping_rounds': parameters.get('early_stopping_rounds', 10),
        'tree_method': parameters.get('tree_method', 'auto'),
        'booster': parameters.get('booster', 'gbtree')
    }

    # Make objective, num_class and eval_metric consistent with the number of classes
    # (xgboost fails on 'binary:*' objective with num_class, and on binary metrics
    # 'error'/'logloss' with 'multi:*' objectives and vice versa)
    if len(class_names) == 2:
        xgb_params['objective'] = parameters.get('objective', 'binary:logistic')

    if len(class_names) > 2 and str(xgb_params['objective']).startswith('binary:'):
        logger.warning(f"XGBoost: objective '{xgb_params['objective']}' is binary, but there are "
                       f"{len(class_names)} classes - using 'multi:softprob'")
        xgb_params['objective'] = 'multi:softprob'

    if str(xgb_params['objective']).startswith('binary:'):
        # Binary objective: no num_class, binary metrics
        xgb_params.pop('num_class', None)
        metric_map = {'mlogloss': 'logloss', 'merror': 'error'}
    else:
        # Multi-class objective (also valid for 2 classes): multi-class metrics
        metric_map = {'logloss': 'mlogloss', 'error': 'merror'}
    xgb_params['eval_metric'] = metric_map.get(xgb_params['eval_metric'], xgb_params['eval_metric'])

    # Enhanced cross-validation with metrics
    cv_scores = []
    cv_predictions = []  # Store predictions from each fold
    cv_true_labels = []  # Store true labels from each fold
    cv_class_reports = []  # Store classification reports from each fold
    cv_confusion_matrices = []  # Store confusion matrices for each fold (using overall classes)
    cv_feature_importances = []  # Store feature importances from each fold

    # Sensitivity and specificity tracking for binary classification
    cv_sensitivity_scores = []  # Store sensitivity from each fold (for binary)
    cv_specificity_scores = []  # Store specificity from each fold (for binary)

    if cv_method == 'stratified':
        cv = StratifiedKFold(n_splits=cv_folds, shuffle=True, random_state=xgb_params['random_state'])
    elif cv_method == 'group':
        groups = parameters.get('groups')
        if groups is None:
            raise ValueError("Groups parameter required for group cross-validation")
        cv = GroupKFold(n_splits=cv_folds)
    else:
        # Simple K-fold
        from sklearn.model_selection import KFold
        cv = KFold(n_splits=cv_folds, shuffle=True, random_state=xgb_params['random_state'])

    # For group CV, we need to handle groups in the loop
    groups_data = parameters.get('groups') if cv_method == 'group' else None

    for fold_idx, (train_idx, test_idx) in enumerate(cv.split(X_processed, y_encoded, groups=groups_data)):
        X_train, X_test = X_processed[train_idx], X_processed[test_idx]
        y_train, y_test = y_encoded[train_idx], y_encoded[test_idx]
        y_test_original = y[test_idx]

        # Train and predict. Early stopping uses a validation part of the
        # training rows only: the test fold must not choose the number of trees.
        xgb_cv = xgb.XGBClassifier(**xgb_params)
        _fit_xgb_with_early_stopping(
            xgb_cv, X_train, y_train,
            early_stopping_rounds=xgb_params.get('early_stopping_rounds', 10),
            random_state=xgb_params['random_state'],
            groups=groups_data[train_idx] if groups_data is not None else None
        )

        # Predict probabilities and classes
        y_pred_encoded = xgb_cv.predict(X_test)

        # Ensure y_pred_encoded is 1D array
        if y_pred_encoded.ndim > 1:
            # If it's 2D, convert to 1D by taking argmax
            if y_pred_encoded.shape[1] > 1:
                y_pred_encoded = np.argmax(y_pred_encoded, axis=1)
            else:
                y_pred_encoded = y_pred_encoded.flatten()

        # Convert back to original class labels
        y_pred = le.inverse_transform(y_pred_encoded)

        # Store fold results
        fold_accuracy = accuracy_score(y_test_original, y_pred)
        cv_scores.append(fold_accuracy)
        cv_predictions.extend(y_pred)
        cv_true_labels.extend(y_test_original)

        # Store per-fold classification report
        fold_report = classification_report(y_test_original, y_pred, output_dict=True, zero_division=0)
        cv_class_reports.append(fold_report)

        # Create confusion matrix using overall classes to ensure consistent dimensions
        cm = confusion_matrix(y_test_original, y_pred, labels=overall_classes_list)
        cv_confusion_matrices.append(cm)

        # Calculate sensitivity and specificity for binary classification using the full 2x2 matrix
        if n_overall_classes == 2:
            tn, fp, fn, tp = cm.ravel()
            sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
            specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
            cv_sensitivity_scores.append(sensitivity)
            cv_specificity_scores.append(specificity)
        else:
            # For multiclass, we can't calculate single sensitivity/specificity values per fold
            cv_sensitivity_scores.append(None)
            cv_specificity_scores.append(None)

        # Store feature importances from this fold
        if hasattr(xgb_cv, 'feature_importances_'):
            cv_feature_importances.append(xgb_cv.feature_importances_)

    # Calculate CV-based metrics
    cv_precision_scores = []
    cv_recall_scores = []
    cv_f1_scores = []

    # Track if this is binary classification
    is_binary = n_overall_classes == 2

    for report in cv_class_reports:
        # Extract macro averages from each fold
        if 'macro avg' in report:
            cv_precision_scores.append(report['macro avg']['precision'])
            cv_recall_scores.append(report['macro avg']['recall'])
            cv_f1_scores.append(report['macro avg']['f1-score'])

    # Calculate overall CV confusion matrix (aggregated)
    cv_confusion_matrix_aggregated = confusion_matrix(cv_true_labels, cv_predictions,
                                                      labels=overall_classes_list).tolist()

    # Calculate overall CV sensitivity and specificity from aggregated confusion matrix
    if is_binary:
        cm_agg = np.array(cv_confusion_matrix_aggregated)
        tn, fp, fn, tp = cm_agg.ravel()
        cv_sensitivity_aggregated = tp / (tp + fn) if (tp + fn) > 0 else 0
        cv_specificity_aggregated = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        cv_sensitivity_aggregated = None
        cv_specificity_aggregated = None

    # Calculate average confusion matrix using the outer function
    if cv_confusion_matrices:
        cv_confusion_matrix_avg, avg_matrix_classes = create_average_confusion_matrix(
            cv_confusion_matrices, [overall_classes_list] * len(cv_confusion_matrices)
        )

        # Calculate average sensitivity and specificity from averaged confusion matrix (binary only)
        if is_binary:
            cm_avg = np.array(cv_confusion_matrix_avg)
            tn, fp, fn, tp = cm_avg.ravel()
            cv_sensitivity_avg = tp / (tp + fn) if (tp + fn) > 0 else 0
            cv_specificity_avg = tn / (tn + fp) if (tn + fp) > 0 else 0
        else:
            cv_sensitivity_avg = None
            cv_specificity_avg = None
    else:
        cv_confusion_matrix_avg = []
        avg_matrix_classes = []
        cv_sensitivity_avg = None
        cv_specificity_avg = None

    # Fit final model on all data
    xgb_final = xgb.XGBClassifier(**xgb_params)

    # Early stopping on a 10 % validation part (grouped by sample for Group
    # K-Fold); without early stopping, or if no validation part can be made,
    # the model is fitted on all rows.
    _fit_xgb_with_early_stopping(
        xgb_final, X_processed, y_encoded,
        early_stopping_rounds=xgb_params.get('early_stopping_rounds', 10),
        random_state=xgb_params['random_state'],
        groups=groups_data
    )

    # Predict on all data
    y_pred_encoded_full = xgb_final.predict(X_processed)

    # Ensure y_pred_encoded_full is 1D array
    if y_pred_encoded_full.ndim > 1:
        # If it's 2D, convert to 1D by taking argmax
        if y_pred_encoded_full.shape[1] > 1:
            y_pred_encoded_full = np.argmax(y_pred_encoded_full, axis=1)
        else:
            y_pred_encoded_full = y_pred_encoded_full.flatten()

    y_pred_proba_full = xgb_final.predict_proba(X_processed)
    y_pred_full = le.inverse_transform(y_pred_encoded_full)

    # Get feature importances
    feature_importances = None
    if hasattr(xgb_final, 'feature_importances_'):
        feature_importances = xgb_final.feature_importances_.tolist()

        # Calculate average feature importances across CV folds
        if cv_feature_importances:
            avg_cv_feature_importances = np.mean(cv_feature_importances, axis=0).tolist()
        else:
            avg_cv_feature_importances = None

    # Get the final model's class order
    final_classes = class_names

    # Reorder the average confusion matrix to match final_classes if needed
    if cv_confusion_matrix_avg and avg_matrix_classes != final_classes:
        # Create mapping from avg_matrix_classes to final_classes
        avg_to_final = {cls: idx for idx, cls in enumerate(avg_matrix_classes)}

        # Create reordered confusion matrix
        n_classes = len(final_classes)
        reordered_matrix = np.zeros((n_classes, n_classes))

        for i, true_cls in enumerate(final_classes):
            for j, pred_cls in enumerate(final_classes):
                if true_cls in avg_to_final and pred_cls in avg_to_final:
                    orig_i = avg_to_final[true_cls]
                    orig_j = avg_to_final[pred_cls]
                    if (orig_i < len(cv_confusion_matrix_avg) and
                            orig_j < len(cv_confusion_matrix_avg[0])):
                        reordered_matrix[i, j] = cv_confusion_matrix_avg[orig_i][orig_j]

        cv_confusion_matrix_avg = reordered_matrix.tolist()
        avg_matrix_classes = final_classes

    # Calculate sensitivity and specificity for full model (self-test)
    full_confusion_matrix = confusion_matrix(y, y_pred_full, labels=final_classes)
    if is_binary and full_confusion_matrix.shape == (2, 2):
        tn, fp, fn, tp = full_confusion_matrix.ravel()
        full_sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0
        full_specificity = tn / (tn + fp) if (tn + fp) > 0 else 0
    else:
        full_sensitivity = None
        full_specificity = None

    # Validate consistency between metrics
    validation_message = None
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        avg_sens_spec = (full_sensitivity + full_specificity) / 2
        accuracy = accuracy_score(y, y_pred_full)
        accuracy_diff = abs(accuracy - avg_sens_spec)
        if accuracy_diff > 0.01:  # Allow small floating point differences
            validation_message = f"Note: Accuracy ({accuracy:.3f}) differs from (sensitivity + specificity)/2 ({avg_sens_spec:.3f}) by {accuracy_diff:.3f}"

    # Calculate CV sensitivity mean and std (filtering out None values)
    cv_sensitivity_scores_filtered = [s for s in cv_sensitivity_scores if s is not None]
    if cv_sensitivity_scores_filtered:
        cv_sensitivity_mean = np.mean(cv_sensitivity_scores_filtered)
        cv_sensitivity_std = np.std(cv_sensitivity_scores_filtered)
    else:
        cv_sensitivity_mean = None
        cv_sensitivity_std = None

    # Calculate CV specificity mean and std (filtering out None values)
    cv_specificity_scores_filtered = [s for s in cv_specificity_scores if s is not None]
    if cv_specificity_scores_filtered:
        cv_specificity_mean = np.mean(cv_specificity_scores_filtered)
        cv_specificity_std = np.std(cv_specificity_scores_filtered)
    else:
        cv_specificity_mean = None
        cv_specificity_std = None

    cv_confusion_matrices_lists = [cm.tolist() for cm in cv_confusion_matrices]

    # Get additional XGBoost-specific information
    best_iteration = getattr(xgb_final, 'best_iteration', xgb_params.get('n_estimators', 100))
    best_score = getattr(xgb_final, 'best_score', None)

    results = {
        'method': 'XGBoost',
        'cv_scores': cv_scores,
        'cv_mean': np.mean(cv_scores) if cv_scores else 0.0,
        'cv_std': np.std(cv_scores) if cv_scores else 0.0,
        'cv_precision': np.mean(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_precision_std': np.std(cv_precision_scores) if cv_precision_scores else 0.0,
        'cv_recall': np.mean(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_recall_std': np.std(cv_recall_scores) if cv_recall_scores else 0.0,
        'cv_f1': np.mean(cv_f1_scores) if cv_f1_scores else 0.0,
        'cv_f1_std': np.std(cv_f1_scores) if cv_f1_scores else 0.0,
        # Sensitivity metrics
        'cv_sensitivity': cv_sensitivity_mean,
        'cv_sensitivity_std': cv_sensitivity_std,
        'cv_sensitivity_aggregated': cv_sensitivity_aggregated,
        'cv_sensitivity_avg': cv_sensitivity_avg,
        'cv_sensitivity_scores': cv_sensitivity_scores,  # Per-fold sensitivities
        # Specificity metrics
        'cv_specificity': cv_specificity_mean,
        'cv_specificity_std': cv_specificity_std,
        'cv_specificity_aggregated': cv_specificity_aggregated,
        'cv_specificity_avg': cv_specificity_avg,
        'cv_specificity_scores': cv_specificity_scores,  # Per-fold specificities
        # Validation flag
        'is_binary': is_binary,
        'validation_message': validation_message,
        # Full model metrics
        'sensitivity': full_sensitivity,
        'specificity': full_specificity,
        # Existing metrics
        'cv_confusion_matrix': cv_confusion_matrix_avg,
        'cv_confusion_matrices': cv_confusion_matrices_lists,
        'cv_confusion_matrix_aggregated': cv_confusion_matrix_aggregated,
        'cv_class_reports': cv_class_reports,
        'classification_report': classification_report(y, y_pred_full, output_dict=True, zero_division=0),
        'confusion_matrix': full_confusion_matrix.tolist(),
        'feature_importances': feature_importances,
        'cv_feature_importances': avg_cv_feature_importances if 'avg_cv_feature_importances' in locals() else None,
        'labels': y.tolist(),
        'predictions': y_pred_full.tolist(),
        'prediction_probabilities': y_pred_proba_full.tolist() if hasattr(xgb_final, 'predict_proba') else None,
        'classes': final_classes,
        'xgboost_parameters': {k: v for k, v in xgb_params.items() if k != 'early_stopping_rounds'},
        'early_stopping_rounds': xgb_params.get('early_stopping_rounds', 10),
        'best_iteration': best_iteration,
        'best_score': best_score if best_score is not None else None
    }

    # Add metric consistency check for binary classification
    if is_binary and full_sensitivity is not None and full_specificity is not None:
        results['metric_consistency_check'] = {
            'accuracy': accuracy_score(y, y_pred_full),
            'sensitivity': full_sensitivity,
            'specificity': full_specificity,
            'avg_sens_spec': (full_sensitivity + full_specificity) / 2,
            'difference': abs(accuracy_score(y, y_pred_full) - (full_sensitivity + full_specificity) / 2)
        }

    # Add feature selection info if used
    if feature_selection_params:
        results['feature_selection'] = {
            'selected_features': selected_features,
            'feature_scores': feature_scores,
            'method': feature_selection_params.get('method', 'anova'),
            'k': feature_selection_params.get('k', 'all')
        }

    # Final fitted model for inference on unknown samples (popped in run_analysis)
    return attach_model_bundle(results, xgb_final, selected_features, feature_selection_params, label_encoder=le)

Limitations and common pitfalls

• Early stopping on small data. See Early stopping and validation.
• Small data. With tens of samples, deep trees (max_depth = 6) and 100 rounds overfit easily. Lower Max Depth (2–3), raise Min Child Weight, or use fewer trees.
• No permutation test. The permutation plot is not available for XGBoost.

References

• Chen T., Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785–794 (2016). doi:10.1145/2939672.2939785
• Friedman J. H. Greedy function approximation: a gradient boosting machine. Annals of Statistics, 29, 1189–1232 (2001). doi:10.1214/aos/1013203451
• XGBoost documentation: Introduction to boosted trees.