Confidence Intervals and the Margin of Error: Built From Four Numbers

1. What an interval claims, in words that survive scrutiny

Before the definition: you have sixteen orders averaging 618.13. Somebody asks what the average order is for the business as a whole. Say what is wrong with answering "618.13", before reading on.

What is wrong is that 618.13 is the average of the sixteen orders you happen to have, and nobody asked about those sixteen. They asked about the business, and your sixteen are a sample of it. A different sixteen would have produced a different number, so quoting this one to two decimal places claims a precision you do not have.

A confidence interval replaces the single number with a range plus a statement about how often the method gets it right. The careful wording is: if you repeated this whole exercise many times, drawing a fresh sample each time and building an interval the same way, 95 percent of those intervals would contain the true value. The picture at the top of this page is nineteen out of twenty.

Notice what that sentence puts the 95 percent on. It is a property of the procedure, across many samples. It is not a property of the one interval sitting in front of you, which either contains the true value or does not, and you will never know which.

That distinction sounds pedantic and it is the reason the next four sections exist. In practice you can read an interval as "the range of values this data is consistent with", and you will get almost everything right. It is when you start attaching probabilities to the one interval you have that the errors begin.

2. Building one from four numbers

Before the arithmetic: you have a mean of 618.125, a standard deviation of 236.81, and sixteen rows. Guess which of those three the interval's width depends on, before reading on.

The last two. The mean sets where the interval sits; the spread and the count set how wide it is. Four steps.

1. The standard error. Divide the standard deviation by the square root of the count: 236.81 ÷ √16 = 236.81 ÷ 4 = 59.20.
2. The multiplier. For 95 percent confidence with 16 rows, it is 2.131. Section three explains where it comes from and why it is not 1.96.
3. The margin of error. Multiply: 2.131 × 59.20 = 126.19.
4. The interval. Put it either side of the mean: 618.125 − 126.19 = 491.94 and 618.125 + 126.19 = 744.31.

So the answer is 618.13, with a 95 percent interval from 491.94 to 744.31. Say out loud what that width tells you about the sixteen rows before reading on. It says they are not many. A quarter-of-a-million-row table with the same spread would have produced an interval about two dollars wide.

from scipy import stats
import numpy as np

x  = df['Revenue']
m  = x.mean()                       # 618.125
se = x.std(ddof=1) / np.sqrt(len(x)) # 59.2028
stats.t.interval(0.95, len(x) - 1, loc=m, scale=se)
# (491.937, 744.313)

' Excel
=AVERAGE(Orders[Revenue])                                      ' 618.125
=CONFIDENCE.T(0.05, STDEV.S(Orders[Revenue]), COUNT(Orders[Revenue]))
' 126.188  -> the margin of error, add and subtract it yourself

-- SQL gives you the ingredients; do the last step in the reporting layer
SELECT COUNT(*)                                     AS n,
       AVG(Revenue)                                 AS mean_order,
       STDDEV_SAMP(Revenue)                         AS sd,
       STDDEV_SAMP(Revenue) / SQRT(COUNT(*))        AS standard_error
FROM Orders;

3. Why the standard error is not the standard deviation

Before the distinction: our standard deviation is 236.81 and our standard error is 59.20. Both are in dollars and both describe uncertainty. Say what each is uncertain about, before reading on.

The standard deviation describes how spread out the individual orders are. Orders vary from 240 to 1,100, and 236.81 is roughly the typical distance from the middle. Collect ten thousand more orders and this number will not shrink, because the orders themselves are not becoming more alike.

The standard error describes how precisely you have pinned down the average. It answers a different question: if I ran this whole sampling exercise again, how much would my computed average jump around? Collect ten thousand more orders and this number does shrink, because averages of large samples are stable even when the underlying values are not.

The division by the square root of n is what turns one into the other, and it is the single most useful piece of arithmetic in applied statistics. It is why "we surveyed 1,000 people" gives a margin of error around 3 percentage points whether the population is a city or a country. The population size barely matters; the sample size is what buys precision.

Now the multiplier. If you knew the true spread of the population you would use 1.96, the value that cuts off the middle 95 percent of a bell curve. You do not know it, you estimated it from the same sixteen rows, and that extra uncertainty has to be paid for. The t distribution charges you for it by widening the multiplier, and the charge depends on how many rows you had.

On 16 rows the t multiplier costs you 9 percent of width. On 1,000 it is a rounding error, which is why the "multiply by 2" shortcut in the opening paragraph is fine on any real-sized table and slightly optimistic on a tiny one.

4. Reading it, and the three readings that are wrong

Before the list: you have an interval of 491.94 to 744.31. Write down what you would say it means, in one sentence, before reading mine. Most people's first sentence is one of the three below.

The second one is worth dwelling on because it is the most consequential in day-to-day work. Our interval is 252 wide and our data spans 860. An interval on the mean gets narrower as you collect more data; the spread of the actual values does not. Confusing them means telling a warehouse to plan for orders between 492 and 744 when a third of orders fall outside that range.

What I would actually write: "Average order value was 618, with a 95 percent interval from 492 to 744, on 16 orders. Individual orders ranged from 240 to 1,100." Two sentences, and neither of them can be misread.

5. The two width dials: confidence level and sample size

Before the numbers: you want a narrower interval. Name the two things you could change, and say which one is honest.

You can lower the confidence level, or you can collect more data. Only the second one buys anything.

Dropping to 90 percent narrows the interval by 22 points either side, and it does so by accepting that the method now misses one time in ten instead of one in twenty. Nothing was learned. The data did not improve. You simply agreed to be wrong more often, which is a legitimate choice when the cost of being wrong is low and a bad one when you did it because the chart looked untidy.

Sample size is the dial that actually works, and it works on a square root.

Four times the data roughly halves the interval. Not four times narrower, half. That is the single most important planning fact on this page, because it tells you what a data-collection request is worth before you make it. Going from 16 rows to 64 is transformative. Going from 1,024 to 4,096 buys you seven dollars of precision, and it is very unlikely anybody needs that.

Now picture the last estimate you were asked to "firm up". Would four times the data have changed the decision it fed, or was the interval already narrow enough to act on?

6. Intervals for a percentage, and where the simple formula breaks

Before the arithmetic: 7 of our 16 orders are for desks, which is 43.75 percent. Predict roughly how wide the interval on that percentage is, before reading on.

Very wide, and the shape of the calculation is the same as before. For a proportion, the standard error is the square root of p times one minus p, divided by n.

p  = 7 / 16 = 0.4375
se = sqrt(0.4375 × 0.5625 / 16) = sqrt(0.015381) = 0.12402
95% interval = 0.4375 ± 1.96 × 0.12402 = 0.1944 to 0.6806

So the desk share is somewhere between 19 and 68 percent. On sixteen orders, "roughly 44 percent" carries essentially no information, and the interval is what makes that visible rather than leaving the reader to assume the number is solid.

That simple formula, called the Wald interval, has a specific failure worth knowing. Watch what happens when every order is a desk.

p  = 16 / 16 = 1.0
se = sqrt(1.0 × 0.0 / 16) = 0
95% interval = 1.0 to 1.0

It claims perfect certainty that 100 percent of all orders, forever, are desks, on the basis of sixteen rows. The formula does not merely become imprecise near 0 and 1; it collapses. The same thing happens more subtly whenever your counts are small, and the usual rule of thumb is that you need at least 5 successes and 5 failures before trusting it. Our 7 and 9 clear that bar by very little.

The fix is a better formula called the Wilson interval, which every statistics library provides and which behaves sensibly at the edges. On our desk share it gives 23.1 to 66.8 percent, a little narrower and shifted toward the middle.

from scipy import stats
stats.binomtest(7, 16).proportion_ci(0.95, method='wilson')
# ConfidenceInterval(low=0.2310, high=0.6682)

Use Wilson by default. It costs one argument and it removes an entire category of embarrassing output.

7. Intervals for a difference, and the zero test

Before the technique: you compare two regions and one is 168.50 ahead. Say what an interval on that gap would need to contain for the gap to be unconvincing.

Zero. If an interval on a difference includes zero, then "no difference at all" is one of the values your data is consistent with, and you cannot claim a direction. Here is the comparison from the p-values guide, run as an interval.

South: 4 orders, mean 667.50
North: 5 orders, mean 499.00
Difference: 168.50 in South's favour
95% interval on the difference: −256 to +593

The interval spans zero comfortably. The data is consistent with South being ahead by 593 and with North being ahead by 256, and picking the middle of that range as the answer would be an act of imagination. That is the same conclusion the p-value reached, and the interval gets there while also telling you how big the effect could plausibly be, which the p-value never does.

This is why I would put an interval in a report where a p-value is optional. A p-value answers "could this be nothing?" An interval answers that plus "and if it is something, how big?" The second question is almost always the one that was actually asked.

One warning, repeated from section four because it is the most common mistake in reading charts: do not test two separate intervals for overlap. Build the interval on the difference. Two 95 percent intervals can overlap slightly and still correspond to a difference that clears any reasonable bar, because the uncertainty in a difference is not the sum of the two uncertainties.

Edge cases that catch people out

Six that each cost somebody an afternoon.

An interval on the whole population. If your table is every order that exists, the average is the average. There is nothing to infer and no interval to compute. Intervals answer "how far off could my sample be", and a census has no sample.

Non-random samples. Every formula here assumes your rows are a random draw. Last month's orders are not a random sample of all orders, survey respondents are not a random sample of customers, and no interval width fixes that. The interval measures sampling noise only, and it says nothing about the bias in how the rows were chosen.

Heavily skewed data on small samples. The t interval assumes the sample mean behaves roughly like a bell curve, which it does on 30-plus rows for most shapes. On 10 rows of something like income, it will not, and the interval is optimistic. Bootstrap it instead, or report percentiles.

Reporting the interval and then ignoring it. If your recommendation would flip somewhere inside your own interval, you do not have a recommendation yet. That is the most useful thing an interval does, and it is the easiest to walk past.

Confusing it with a prediction interval. Ours says where the average lies. A prediction interval says where the next single order will lie, and it is much wider because it carries the spread of individual orders as well as the uncertainty in the mean.

Intervals on tiny groups in a grouped report. Every group gets its own n, so West's three orders produce an interval so wide it is almost useless. Print the count beside every interval, the same discipline GROUP BY and HAVING asks for.