Feature selection

Feature selection keeps only the $k$ feature columns that are most related to the label and drops the rest before the model is built. Use it when you have many colour channels and suspect that most of them carry no information about the class.

It is available for all classification methods. PCA ignores it.

Settings

In the analysis dialog, turn on Enable Feature Selection, then set:

UI label	Parameter	Default	Notes
Enable Feature Selection	`feature_selection.enabled`	off	When off, all features are used, whatever the other fields say.
Selection Method	`feature_selection.method`	ANOVA F-test (`anova`)	See Methods.
Number of Features	`feature_selection.k`	empty (`'all'`)	Empty means "keep all features", which is the same as turning selection off.

Methods

Both methods score every feature separately, against the label, and keep the $k$ features with the highest scores. The selector is sklearn.feature_selection.SelectKBest with one of two score functions.

UI label	`method`	Score function	What it measures
ANOVA F-test	`anova`	`f_classif`	Ratio of the variance between class means to the variance within classes. High when the class means of the feature are far apart compared with the spread inside each class. Detects linear (mean) differences only.
Mutual Information	`mutual_info`	`mutual_info_classif`	How much knowing the feature value reduces the uncertainty about the class. Also detects non-linear relations, but is noisier on small data sets.

For one feature with values $x_{ij}$ (sample $j$ of class $i$ ), $K$ classes and $n$ samples, the ANOVA F statistic is

F = \frac{\sum_i n_i (\bar{x}_i - \bar{x})^2 / (K - 1)} {\sum_i \sum_j (x_{ij} - \bar{x}_i)^2 / (n - K)} .

The scores of all features are saved in the results (feature_selection.feature_scores), together with the indices of the kept features (feature_selection.selected_features).

Scoring is univariate: two features that are useless alone but useful together can both be dropped, and several almost identical features can all be kept.

Number of features (k)

$k$ larger than the number of feature columns is reduced to the number of columns.
$k$ is not limited by the number of samples. Most methods (PLS-DA, SIMCA, random forest, …) work when there are more features than samples.
If selection fails for any reason, the analysis continues with all features.

There is no automatic choice of $k$ . If you compare several values of $k$ by their CV accuracy and pick the best one, the chosen accuracy is optimistic: you have tuned $k$ on the same folds that report the result.

Feature selection inside cross-validation

warning

Features are selected once, on all rows, before cross-validation. The labels of the test folds are therefore used to choose the features, and the cross-validated metrics are optimistic. The effect is small when $k$ is close to the number of features and large when you keep a few features out of many.

To check how much of the result is real, run a permutation test with Feature Selection in Permutations set to Repeated for each permutation (recommended). Then every shuffled model goes through the same selection as the original one, and the p-value is fair. See also Cross-validation and data leakage.

Source code

The selector:

analysis.py · select_features() · lines 72–141
def select_features(X, y, feature_selection_params, problem_type='classification'):
    """
    Utility function for feature selection using ANOVA F-test or other methods.

    Parameters:
    -----------
    X : array-like, shape (n_samples, n_features)
        Input data
    y : array-like, shape (n_samples,)
        Target values
    feature_selection_params : dict
        Parameters for feature selection:
        - method: 'anova' (default), 'mutual_info', etc.
        - k: number of features to select (int or 'all')
        - threshold: threshold for feature selection (for variance-based methods)
    problem_type : str
        'classification' or 'regression'

    Returns:
    --------
    X_selected : array-like
        Data with selected features
    selected_features : list
        Indices of selected features
    feature_scores : array-like
        Scores for all features
    """
    if feature_selection_params is None:
        return X, list(range(X.shape[1])), None

    method = feature_selection_params.get('method', 'anova')
    k = feature_selection_params.get('k', 'all')
    threshold = feature_selection_params.get('threshold', 0)

    # If k is 'all', no feature selection
    if k == 'all':
        return X, list(range(X.shape[1])), None

    # k cannot exceed the number of available features. It is deliberately
    # not limited by the number of samples: p > n is valid for most methods
    # (PLS-DA, SIMCA, RF, ...) and the user's k must not be changed silently.
    n_features = X.shape[1]
    if isinstance(k, int) and k > n_features:
        k = n_features

    if problem_type == 'classification':
        if method == 'anova':
            selector = SelectKBest(score_func=f_classif, k=k)
        elif method == 'mutual_info':
            from sklearn.feature_selection import mutual_info_classif
            selector = SelectKBest(score_func=mutual_info_classif, k=k)
        else:
            selector = SelectKBest(score_func=f_classif, k=k)
    else:  # regression
        if method == 'anova':
            selector = SelectKBest(score_func=f_regression, k=k)
        elif method == 'mutual_info':
            from sklearn.feature_selection import mutual_info_regression
            selector = SelectKBest(score_func=mutual_info_regression, k=k)
        else:
            selector = SelectKBest(score_func=f_regression, k=k)

    try:
        X_selected = selector.fit_transform(X, y)
        selected_features = selector.get_support(indices=True).tolist()
        feature_scores = selector.scores_.tolist() if hasattr(selector, 'scores_') else None
        return X_selected, selected_features, feature_scores
    except Exception as e:
        print(f"Feature selection failed: {e}. Using all features.")
        return X, list(range(X.shape[1])), None

How the dialog setting is read (the Enable Feature Selection checkbox):

analysis.py · get_feature_selection_params() · lines 151–161
def get_feature_selection_params(parameters):
    """
    parameters['feature_selection'], or None when feature selection is off:
    missing/empty, or explicitly disabled with enabled=False (the
    "Enable Feature Selection" checkbox). A dict without the 'enabled' key
    (API / older clients) counts as on.
    """
    fs = (parameters or {}).get('feature_selection')
    if not fs or fs.get('enabled') is False:
        return None
    return fs

Where it is called, in every run_<method>_analysis function before the CV loop. Example from LDA:

analysis.py · run_lda_analysis() · lines 1006–1014
feature_selection_params = get_feature_selection_params(parameters)
if feature_selection_params:
    X_processed, selected_features, feature_scores = select_features(
        X, y, feature_selection_params, 'classification'
    )
else:
    X_processed = X
    selected_features = list(range(X.shape[1]))
    feature_scores = None

References

Guyon I., Elisseeff A. An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157–1182 (2003). PDF
Kraskov A., Stögbauer H., Grassberger P. Estimating mutual information. Physical Review E, 69, 066138 (2004). doi:10.1103/PhysRevE.69.066138
scikit-learn user guide: Univariate feature selection.

Settings​

Methods​

Number of features (k)​

Feature selection inside cross-validation​

Source code​

References​