Wiki
Core11 min read

Feature generation

A model can only see the coordinates you give it. Good features move the signal into a space the model can use.

A linear model is a fixed-shape ruler: it can only draw straight lines through the features you hand it. If the truth is a curve, no amount of training will help — until you add the square of a feature, at which point the same model suddenly fits. That is the whole promise of feature engineering: it changes the space the model searches, and the right change can be worth more than a fancier algorithm.

The demo below fits a real polynomial regression by least squares. On the raw feature the curve is invisible; add x2x^2 and the fit snaps into place. The coefficients and R2R^2 are solved from the normal equations, so you can watch the residual collapse as the basis grows.

A linear model on the raw feature underfits — add polynomial features and refit

Engineered basis

Fit quality

R² = 0.630

bias = 0.92x = -0.86

This is a hand-built polynomial regression, not a neural network, and the target is synthetic — the point is the mechanism. Feature engineering changes the space the model can represent; a model that looked useless on raw x becomes a good fit once x² exists. Too high a degree and the curve starts chasing noise, which is the mirror-image failure.

Features encode assumptions

Every transformation is a hypothesis about the problem. A log transform says the effects are multiplicative. An interaction term says one feature matters more depending on another. Binning says the response is flat within a range. Making the assumption explicit is the point — a good feature is a piece of domain knowledge, written in a form the model can use.

The standard transformations

  • Numeric rescaling. Standardisation, z=(x−μ)/σz = (x - \mu)/\sigma, and min-max scaling put features on comparable scales. This matters enormously for distance-based and regularised models, and not at all for trees.
  • Non-linear warps. log⁡\log, ⋅\sqrt{\cdot}, and reciprocal transforms compress heavy tails. Monetarily, the log of income is often more linear in other variables than income itself.
  • Interactions and polynomials. Products xixjx_i x_j and powers xikx_i^k let a linear model represent curvature and conditional effects.
  • Categorical encoding. One-hot for low cardinality, target/mean encoding or learned embeddings for high cardinality — each with its own leakage trap.
  • Binning. Equal-width or quantile bins trade resolution for robustness, and can turn a noisy continuous variable into a stable ordinal one.
  • Temporal features. Hour of day, day of week, time since last event, and cyclical encodings (sin⁡\sin and cos⁡\cos of the phase) that avoid the artificial discontinuity between 23:00 and 00:00.

The leakage boundary

The single most common way feature engineering fails is target leakage: a feature that encodes the answer and will not be available at prediction time. The customer's "retention offer accepted" flag is the churn outcome in disguise; the "fraud confirmed" field is the label. Leakage shows up as a model that validates beautifully and dies in production. The test is mechanical: for every feature, ask whether its value could be observed before the event you are predicting, in the system that will serve the prediction.

Careful

Fit every transformation inside the cross-validation fold. If you standardise, impute, or target-encode on the whole dataset and then split, the validation score is contaminated by the test rows. Pipelines exist precisely so that preprocessing is refit on each training fold — a detail that separates an honest score from a flattering one.

Illustrative vs real

The artifact is a two-variable polynomial fit on synthetic points — a demonstration of a mechanism, not a feature-selection study. In real work the candidate space is huge, the target is noisy, and the discipline that pays is not inventing many features but validating honestly that each one earns its place.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, Features & recommenders

0 / 4 answered

  1. 1A feature x is standardised to z = (x − μ)/σ. The mean of z over the training data is 0. What is the standard deviation of z (population form)?
    Numeric answer
  2. 2Which feature is most likely to be target leakage for a churn model scored at the moment a customer calls?
    Multiple choice
  3. 3Tree-based models are unaffected by monotonic feature scaling.
    True / false
  4. 4Which pair of transforms encodes a cyclical feature such as hour-of-day without an artificial boundary at midnight?
    Short answer

From the assignment paper

Modeled on NITJ AI-505, Assignment/Quiz

0 / 5 answered

  1. 1A feature ranges from 20 to 120. After min–max scaling onto the unit interval, what value does the observation 45 take?
    Numeric answer
  2. 2Which of these is not a feature-selection method?
    Multiple choice
  3. 3The primary goal of exploratory data analysis is to…
    Multiple choice
  4. 4Which approach replaces a missing value with the average of its column?
    Multiple choice
  5. 5In a linear regression the intercept is the value the line takes when every feature is zero.
    True / false

Where next: feature selection — cutting the candidate set down to the features that actually earn their keep.