Feature selection

Adding a feature can never make a model more wrong on the training set — at worst an unhelpful coordinate gets weight near zero, and at worst it gets weight that happens to fit noise. The damage is on unseen data: irrelevant and redundant features inflate variance, slow training, and make the model harder to explain. Feature selection is the deliberate removal of that tax.

The workbench below makes the distinction concrete. A filter scores each feature independently against the target and takes the top $k$ — fast, but blind to redundancy, so it happily keeps three collinear charge variables. A wrapper evaluates subsets using the model's own score, so it notices that the third correlated feature adds nothing once its siblings are in.

Filters: fast, model-free scores #

A filter computes a statistic per feature and ranks. Common choices:

• Correlation for a numeric target, $\rho(x_j, y)$.
• Mutual information, which catches non-linear dependence that correlation misses: $$I(x; y) = \sum_{x,y} p(x,y)\,\log\frac{p(x,y)}{p(x)\,p(y)}.$$
• Variance / near-zero-constant filters that drop columns carrying almost no information on their own.
• $\chi^2$ or ANOVA F-scores for classification with categorical or continuous inputs respectively.

Filters are $O(p)$ or $O(pn)$ and scale to millions of columns, which is why they are the first pass. Their limitation is structural: they never see a combination, so they cannot remove a feature that is redundant given others.

Wrappers: search, paying for it #

A wrapper treats selection as a search problem over subsets and scores each candidate with cross-validated model performance. The three standard moves:

• Forward selection — start empty, greedily add the feature that improves the score most, stop when no addition helps.
• Backward elimination — start with everything, greedily remove the least useful feature.
• Recursive feature elimination (RFE) — repeatedly fit the model, drop the weakest features, refit.

With $p$ features the search space is $2^p$, so exact best-subset is intractable; greedy methods are heuristics that are good enough in practice. Wrappers are expensive — each evaluation is a model fit — and they overfit the validation score if you are not careful about the split.

Embedded methods: selection during training #

The third family lets the model decide as it fits. Lasso adds an $\ell_1$ penalty that drives coefficients exactly to zero, so the surviving non-zeros are a selection:

$$\hat\beta = \arg\min_\beta \; \lVert y - X\beta\rVert_2^2 + \lambda \lVert \beta \rVert_1.$$

Tree ensembles give impurity- or permutation-based importance; dropout acts as a stochastic form of regularisation that discourages reliance on any single input. Embedded methods sit between filters and wrappers in cost and usually beat both in accuracy per unit of compute.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, Features & recommenders

0 / 4 answered

1. 1Two features each correlate strongly with the target and correlate 0.95 with each other. What should a wrapper typically do?

   A.Keep one and drop the otherB.Keep bothC.Add a third copyD.Drop both

   Check Show answerMultiple choice
2. 2Which feature-selection family scores each feature independently of the model and ignores combinations?

   Check Show answerShort answer
3. 3Recursive feature elimination fits a model repeatedly and drops the weakest features between fits.

   TrueFalse

   Check Show answerTrue / false
4. 4With p = 12 candidate features, how many subsets exist in the full best-subset search space (including the empty set)?

   Check Hint Show answerNumeric answer

From the exam paper

Modeled on NITJ AI-505, End-Sem December 2024

0 / 4 answered

1. 1Principal components are the eigenvectors of the covariance matrix. Which property of that matrix guarantees the eigenvectors for distinct eigenvalues are mutually orthogonal?

   Check Show answerShort answer
2. 2A dataset has 8 numeric features. PCA can drop components, but how many principal components are needed to retain 100% of the total variance?

   Check Hint Show answerNumeric answer
3. 3Which statement best captures the difference between a filter and a wrapper for feature selection?

   A.Filters evaluate feature subsets with the model; wrappers score features independentlyB.Filters and wrappers both evaluate interactions between featuresC.Filters score features independently of the model; wrappers evaluate subsets using the model's own performanceD.Wrappers are always cheaper to run than filters

   Check Show answerMultiple choice
4. 4Because a filter scores each feature on its own, it readily removes a feature that is redundant given the others already selected.

   TrueFalse

   Check Show answerTrue / false

Where next: recommendation systems — feature engineering and similarity combined to fill in the blanks of a ratings matrix.