Wiki
Intro10 min read

Exploratory data analysis

Plots before models: distributions, outliers, and the summary statistics that lie about them.

Four datasets can share the same mean, the same variance, and even the same correlation to two decimal places while being utterly different — Anscombe showed this in 1973 and the lesson has only grown sharper since. The point is not that summary statistics are useless. It is that they are lossy compressions, and compressions are only safe once you know what they threw away.

Exploratory data analysis is the discipline of looking before you compress. Before a single model is fit, you want to know the shape of each variable, how they move together, which points are outliers, and which missingness patterns might be telling you something. EDA is not "quick and dirty" — it is where most real discoveries and most data bugs are found.

Two numbers, two summaries

The mean is dragged around by every value, including the extreme ones; the median only cares about the middle. In a right-skewed distribution — incomes, wait times, request latencies — the mean sits above the median and no single number describes a typical case. When mean and median disagree, that disagreement is information.

Play with the workbench below. The mean chases the tail while the median holds its ground; the box plot exposes outliers the histogram buries; and the bimodal option shows two modes that a single mean would happily average into a value that appears nowhere in the data.

Same dataset, three views — change the shape and watch the summary statistics move

histogram · 16 bins · n = 180

box plot · whiskers extend to 1.5 × IQR, 2 outliers beyond

Distribution

mean
49.66
median
48.88
Q1
42.11
Q3
57.21
IQR
15.10

Note how the mean chases the tail while the median holds its ground in the skewed case, and how the bimodal case shows two modes a single summary statistic would hide. The generator uses a fixed seed so the numbers are reproducible, but they are synthetic — real EDA starts by plotting the data you actually have.

The five-number summary and the box plot

For a sorted sample, the five-number summary is the minimum, the first quartile Q1Q_1, the median Q2Q_2, the third quartile Q3Q_3, and the maximum. The interquartile range is

IQR=Q3−Q1,\mathrm{IQR} = Q_3 - Q_1,

the spread of the middle half of the data. The box plot draws the box from Q1Q_1 to Q3Q_3 with a line at the median, and marks points beyond the fences

Q1−1.5 IQRandQ3+1.5 IQRQ_1 - 1.5\,\mathrm{IQR} \quad \text{and} \quad Q_3 + 1.5\,\mathrm{IQR}

as candidate outliers. The 1.51.5 is a convention, not a law: for a normal distribution it flags roughly 1 in 140 points, so a handful of flags is expected, while a flood of them means the distribution is heavy-tailed and perhaps should be transformed.

Why the histogram depends on the bin width

A histogram estimates a density by counting points per bin. Its appearance depends on the bin width hh: too wide and real modes merge, too narrow and randomness looks like structure. The bias-variance trade-off is explicit — the expected count in a bin of width hh near density f(x)f(x) is approximately nhf(x)n h f(x), so the bin's standard error grows as hh shrinks. There is no single correct width; the honest move is to try several and only believe a feature that survives.

Explore on training data, not everything

Running EDA on the full dataset and then choosing features, thresholds, or a model based on what you saw is a subtle form of leakage. Any decision informed by the test set contaminates it. Reserve the split before you start making choices, and if you must explore globally, keep the decisions general.

Missingness is data too

A missing value is a fact about the process that produced the row. If income is blank more often for one group, dropping those rows silently changes the population your model describes. Always ask: is it missing at random, missing because of an observed variable, or missing because of the value itself? The answer determines whether imputation is safe, and no amount of clever filling fixes a missingness mechanism you refused to name.

Illustrative vs real

The distribution generator in the artifact is synthetic and seeded so numbers are reproducible. Real EDA starts from data you did not design, with units, join keys, and collection quirks that no toy can imitate. The statistics computed here — quartiles, fences, counts — are exactly the ones you would use; only the provenance of the numbers is simplified.

Check yourself

Eduspheria wiki · Data, MLOps & Deployment, Data science foundations

0 / 4 answered

  1. 1A sample has Q1 = 20 and Q3 = 44. What is the upper outlier fence Q3 + 1.5·IQR?
    Numeric answer
  2. 2In a strongly right-skewed distribution, how do the mean and median typically compare?
    Multiple choice
  3. 3Choosing the bin width of a histogram after looking at the data is harmless.
    True / false
  4. 4What is the term for data missing in a way that depends on the unseen value itself, which imputation cannot safely repair?
    Short answer

From the exam paper

Modeled on NITJ AI-505, End-Sem December 2024

0 / 4 answered

  1. 1A NumPy column vector of shape (3, 1) is added to a row vector of shape (1, 4). Broadcasting stretches both to shape (3, 4); how many elements does the result contain?
    Numeric answer
  2. 2In the paper's snippet x is a 3-row, 4-column array. The variable y is np.concatenate((x, x), axis=0). How many rows does y have?
    Numeric answer
  3. 3For the same 3-by-4 array x, the variable z is np.concatenate((x, x), axis=1). How many columns does z have?
    Numeric answer
  4. 4A histogram of the paper's 19 house prices bunches almost every value between 150 and 250, with three isolated spikes far to the right. Using the fence Q3 + 1.5 × IQR, with Q3 = 210 and IQR = 40, how many prices are flagged as high outliers?
    Numeric answer

Where next: statistical inference — turning a sample into a claim about the population, with an honest error bar attached.