Standard Deviation in Everyday Words: What It Measures and How to Read It

1. What the standard deviation actually measures

Before the definition: our sixteen orders average 618.125, and the smallest is 240 while the largest is 1,100. Come up with your own single number for "how far a typical order sits from 618" before reading on. Any reasonable guess will land close.

The honest answer is 236.81, and if you guessed somewhere between 180 and 280 you already understand what the statistic is for. The standard deviation is a single number summarizing how far the values sit from their own average. It is in the same units as the data, so on a revenue column it is measured in dollars, and on the Units column it is measured in units.

That last point is worth pausing on because it is what makes the number usable. An average order of 618 and a standard deviation of 237 can be read as one sentence: orders are typically around 618, give or take a couple of hundred. Nobody needs a statistics background to act on that. What they cannot act on is 618 alone, because 618 alone is equally consistent with every order being 615 and with half the orders being 100 and half being 1,100.

There is a close relative called the variance, which is the standard deviation squared: 56,079.58 here. Variance is the version the mathematics prefers, because squared quantities add up neatly. Standard deviation is the version people can read, because it is back in dollars. Report the standard deviation; the variance is a step on the way to it.

2. Building it by hand from sixteen orders

Before the walkthrough: the obvious way to measure typical distance from the mean is to take each value's distance and average those distances. Try that in your head, adding the distances with their signs, and predict the total before I show it.

The total is exactly zero, and it always will be. That is not a coincidence of this data; it is what the mean is. The mean is the balance point, so everything above it cancels everything below it exactly. Averaging signed distances tells you nothing, ever.

Here is the table, sorted, with each order's distance from 618.125 and that distance squared. Squaring is the standard fix: it throws away the sign, and it does so in a way that keeps big misses expensive.

Four steps, and you have just done three of them.

1. Take each value's distance from the mean. They sum to zero.
2. Square each distance. Now everything is positive. Total: 841,193.75.
3. Divide by 15, one less than the count: 841,193.75 ÷ 15 = 56,079.58. That is the variance.
4. Take the square root, to get back into dollars: √56,079.58 = 236.81.

Step four is the step people forget the reason for. You squared everything in step two, so the units squared too: 56,079.58 is measured in dollars-squared, which is not a thing anybody can picture. The square root undoes the squaring and puts the answer back in dollars.

Every tool does this in one call.

-- SQL
SELECT STDDEV_SAMP(Revenue) AS sd,       -- 236.81, the n-1 version
       STDDEV_POP(Revenue)  AS sd_pop,   -- 229.29, the n version
       VAR_SAMP(Revenue)    AS variance  -- 56,079.58
FROM Orders;

# pandas defaults to n-1
df['Revenue'].std()          # 236.811...
df['Revenue'].std(ddof=0)    # 229.291...

# numpy defaults to n. This catches people out constantly.
np.std(df['Revenue'])            # 229.291...
np.std(df['Revenue'], ddof=1)    # 236.811...

' Excel
=STDEV.S(Orders[Revenue])    ' 236.81, the one you almost always want
=STDEV.P(Orders[Revenue])    ' 229.29

3. Why the divisor is 15 and not 16

Before the explanation: you have sixteen values and you are averaging sixteen squared distances. Say why dividing by sixteen would be the wrong move, before reading on. The clue is in step one of the last section.

The distances were measured from the mean of these same sixteen numbers, not from some outside truth. That mean was itself computed from the data, and it sits exactly where the squared distances come out smallest. Any other number you measured from would give a larger total. So the total of 841,193.75 is the smallest it could possibly be for this data, which means dividing by 16 gives a spread estimate that runs slightly small.

Dividing by 15 corrects for it. The general rule is that you subtract one for each quantity you estimated from the data on the way, and here that is one: the mean. The number of independent pieces of information left over is called the degrees of freedom, and the arithmetic behind the name is simple. Once you know the mean and any fifteen of the values, the sixteenth is forced. Only fifteen of them were free to vary.

On our data the correction is worth 236.81 against 229.29, about three percent. On 16 rows that is small; on 5 rows it is a tenth of the answer; on 5,000 rows it is invisible. Which gives the practical rule:

• Use the n−1 version when your rows are a sample of something larger, which is nearly always. These sixteen orders are some of the orders, not all the orders that will ever exist.
• Use the n version only when your rows genuinely are the entire population and you are describing it rather than inferring from it. Every student in one class, every store you own, this quarter's twelve months exactly.
• When in doubt use n−1, because it is the conservative choice and because on any decent row count the two agree anyway.

The reason this matters more than it should is that the defaults disagree. Excel's STDEV.S, pandas's .std() and SQL's STDDEV_SAMP all use n−1. numpy's np.std() uses n unless you pass ddof=1. Two tools, same data, two different answers, no error message anywhere.

4. Reading the answer: the band, and the z-score

Before the technique: you have a mean of 618.125 and a standard deviation of 236.81. Turn those two numbers into a range you would actually say out loud, then read on.

The move is to add and subtract. 618.125 − 236.81 = 381.31, and 618.125 + 236.81 = 854.94. So one standard deviation spans roughly 381 to 855, and eleven of our sixteen orders sit inside it. That band is the sentence: typical orders run from about 380 to about 855. Two standard deviations spans 144.50 to 1,091.75, and fifteen of sixteen orders fall inside that.

The second move turns a single value into a comparable number. A z-score is how many standard deviations a value sits from the mean: subtract the mean, divide by the standard deviation.

Order 1010, revenue 1100:  (1100 − 618.125) ÷ 236.81 = +2.03
Order 1004, revenue  240:  ( 240 − 618.125) ÷ 236.81 = −1.60

The biggest order is two standard deviations above average and the smallest is one and a half below. Now say what a z-score buys you that the raw numbers do not. It is that a z-score has no units, so it is comparable across columns: an order two standard deviations above average and a delivery time two standard deviations above average are equally unusual for their own columns, even though one is in dollars and the other in days.

That is the whole reason z-scores exist, and it is also the honest limit of them. A z-score says how unusual a value is relative to the spread of its own column. It says nothing about whether the value is wrong, or important, or worth acting on.

5. 68, 95, 99.7, and when that rule does not hold

Before the caveat: you have probably heard that about 68 percent of values fall within one standard deviation. Our data put 11 of 16 inside, which is 68.75 percent. Decide whether that agreement is evidence of anything before reading on.

It is a coincidence, and a useful one to have met. The 68–95–99.7 rule describes the normal distribution, the symmetrical bell-shaped one. Real data is often close enough for the rule to be a fair guide, and our sixteen orders happen to be. But nothing forces it. Revenue, salaries, response times and session lengths are usually piled up at the low end with a long tail to the right, and on that shape a one-standard-deviation band can hold far more or far less than 68 percent.

There is a version of the rule that always holds, with no assumption about shape at all. Chebyshev's inequality says at least 75 percent of any data set is within two standard deviations of its mean, and at least 89 percent within three. It is a guarantee rather than an estimate, which is why the numbers are so much weaker: it has to be true for every possible shape, including the nastiest.

The practical version. Use the 68–95–99.7 rule as a rough sanity check, not as a claim. If you need to state what fraction of your data falls in a range, do not estimate it from the standard deviation at all. Count it, or use percentiles, which describe the data you actually have rather than the shape you assumed it had.

6. Comparing spreads on different scales

Before the problem: the Revenue column has a standard deviation of 236.81 and the Units column has one of 3.70. Which column is more variable? Commit to an answer before reading on, and notice what makes the question awkward.

The question as asked has no answer, because 236.81 dollars and 3.70 units cannot be compared. A standard deviation carries the units of its column, which makes it readable and makes it uncomparable across columns. The same problem shows up between two companies where one reports in dollars and the other in thousands of dollars.

The fix is to divide the standard deviation by the mean, giving the coefficient of variation. It is a ratio, so the units cancel and what is left is spread expressed as a fraction of the average.

Units is the more variable column, and by a clear margin: its spread is 59 percent of its own average against Revenue's 38 percent. That was not visible in the raw standard deviations, and it is the kind of finding that changes what you investigate first.

The same comparison across regions is worth running on any grouped data.

South's orders are the least predictable and West's the most, and neither fact was available from the means. Read the counts, though. A standard deviation computed on three values is barely a measurement, and the caution is the same one that applies to any small group, covered in GROUP BY and HAVING. Two decimal places on a number built from three rows is false precision.

Now picture your own most-used table. Which column would you guess has the highest coefficient of variation, and would you have predicted that from its average alone?

7. What one typo does to it

Before the number: somebody keys order 1013 as 150 lamps instead of 15, so that row's revenue becomes 6,000 instead of 600. The mean goes from 618.125 to 955.625. Predict what happens to the standard deviation.

It goes from 236.81 to 1,365.84, which is nearly six times larger. The reason is step two of the hand calculation: distances get squared, so a value that is ten times too far away contributes a hundred times too much. The squaring that made the arithmetic work is the same squaring that makes the result fragile.

Two consequences follow. First, the standard deviation is even easier to break with one bad row than the mean is, so anything you compute from it inherits that fragility, including z-scores and confidence intervals. Second, and more useful, a standard deviation that jumps when new data arrives is a genuinely good alarm. If last month's spread was 240 and this month's is 1,400 on similar volume, look at the largest few rows before you look at anything else.

If your data has real extremes that are not errors, the standard deviation is the wrong summary of spread and the interquartile range is the right one, because it is built from positions rather than sizes. That is the percentiles and IQR guide, and it is the natural companion to this one.

Edge cases that catch people out

Six that each cost somebody an afternoon.

numpy and pandas disagree by default. np.std(x) divides by n and Series.std() divides by n−1. On 16 rows that is a 3 percent gap between two lines of the same notebook, with nothing to warn you.

The standard deviation of a percentage column is usually meaningless. Averaging rates has the same problem as averaging averages: a 100 percent conversion on 1 visitor and a 5 percent conversion on 10,000 are not two comparable numbers. Compute the spread on the raw counts, not on the derived rate.

A standard deviation on three rows. West's 177.22 above is arithmetically correct and practically noise. Below about 20 rows treat a standard deviation as a hint, and always print the count beside it.

Blanks are skipped, not zeroed. Every tool on this page ignores empty cells, so a column that is one quarter blank reports the spread of the three quarters that answered. Whether the blanks are zeros is a decision, not a default, and handling missing values is where to make it.

Reporting sd on obviously skewed data. If your mean is well above your median, the band goes negative or reaches into territory with no data in it. On house prices you will regularly see mean minus one standard deviation land below zero, which is a signal to switch to percentiles rather than to print it.

Confusing it with standard error. They differ by a division: standard error is the standard deviation divided by the square root of n. The standard deviation describes how spread out your values are, and it does not shrink as you collect more data. The standard error describes how precisely you have pinned down the mean, and it does shrink. Confidence intervals are built on the second one.