What Is a P-Value? Worked by Shuffling Nine Real Orders 126 Ways

1. The question a p-value answers

Before the definition: you compare two regions and one is ahead by 168.50. Name the specific doubt that number leaves you with, in your own words, before reading on.

The doubt is that the gap might be nothing but which orders happened to land where. Split any nine orders into two groups at random and the two group averages will differ, always, by something. So the useful question is not "do they differ" but "is this gap bigger than the gaps random splitting produces?"

A p-value answers exactly that question and nothing beyond it. Formally: assuming there is genuinely no difference between the groups, the p-value is the probability of seeing a gap at least as large as the one you saw. Everything else people say about p-values is a paraphrase that has drifted.

The assumption in the middle of that sentence has a name, the null hypothesis. It is the boring explanation: the regions are the same, the button colour makes no difference, the drug does nothing. The p-value never evaluates your interesting hypothesis. It only measures how uncomfortable your data would be for the boring one.

2. The data: two regions, 168.50 apart

Before the arithmetic: here are nine orders. Look at them and decide, by eye, whether you think South genuinely sells bigger than North, before any statistics.

North (5 orders):  880  240  425  440  510      mean 499.00
South (4 orders):  850  660  280  880           mean 667.50

Difference in means: 667.50 − 499.00 = 168.50

South's average order is 168.50 higher, which is a 34 percent gap and would be a real finding if it held up. Look at the raw values though. North has an 880 and South has a 280, and if you swapped just those two rows between the regions the gap would nearly close. Nine numbers is not many, and that intuition is precisely what the next section makes exact.

3. The shuffle test, all 126 of them

Before the method: if the region label made no difference at all, then any four of these nine orders could equally well have been the South ones. Say how you would use that fact to test the gap, before reading on.

You take the nine order values, forget which region they came from, and deal four of them into a pretend South and five into a pretend North. Compute that split's gap. Do it again for a different split. The gaps you get are what pure chance produces when the label means nothing.

There are exactly 126 ways to choose 4 items from 9, so we do not have to sample. We can do every single one.

import itertools, numpy as np

vals = np.array([880, 240, 425, 440, 510, 850, 660, 280, 880], float)
observed = 667.50 - 499.00        # 168.50

gaps = []
for pick in itertools.combinations(range(9), 4):      # 126 of them
    south = vals[list(pick)]
    north = vals[[i for i in range(9) if i not in pick]]
    gaps.append(south.mean() - north.mean())

gaps = np.array(gaps)
extreme = (np.abs(gaps) >= abs(observed)).sum()
print(len(gaps), extreme, extreme / len(gaps))

126 45 0.35714285714285715

Forty-five of the 126 possible splits produce a gap at least as large as 168.50, in one direction or the other. That is p = 45 ÷ 126 = 0.357, and it is the picture at the top of this page: the darker dots are those 45.

Read the sentence carefully, because it is the whole concept. If the region label meant nothing, we would see a gap this big or bigger about 36 percent of the time. A thing that happens 36 percent of the time by accident is not evidence of anything. It is an ordinary Tuesday.

Note what this calculation did not need. No bell curve, no assumption about the shape of order revenue, no degrees of freedom, no table in the back of a textbook. Just counting. This is called a permutation test, and when the group sizes are small enough to enumerate, it is the most honest version of the idea there is.

Say out loud why the test uses "at least as large" rather than "exactly this large". The reason is that any exact gap is rare; there are 126 splits and most gaps appear only a handful of times. Asking "how often is chance at least this impressive" is the only version of the question that has a stable answer.

4. The t-test: the same answer without the shuffling

Before the comparison: the shuffle needed 126 calculations for nine values. Predict roughly how many it would need for a hundred values per group, and you will see why the classical test exists.

Choosing 100 from 200 gives a number with 59 digits in it. Enumeration stops being possible almost immediately, and the classical tests are shortcuts that get to the same answer using mathematics instead of brute force.

from scipy import stats
stats.ttest_ind(south, north, equal_var=False)
# statistic = 0.9709, pvalue = 0.3692

The t-test gives p = 0.369 against the shuffle's 0.357. The two agree, which is the point: the t-test is not a different concept, it is the same question answered by assuming a shape for the data rather than generating it. When that assumption is reasonable, it is faster and it works on any sample size. When the data is heavily skewed or tiny, the shuffle is the one to trust.

Two practical notes on the function. equal_var=False asks for Welch's version, which does not assume the two groups have the same spread. Make it your default, because it costs almost nothing when the spreads do match and it is the correct answer when they do not. And the test is two-sided by default, meaning it counts gaps in both directions, which matches the abs() in our shuffle code.

Here is the same test in SQL-adjacent form, for when your data lives in a warehouse and you want the ingredients rather than the verdict.

SELECT Region,
       COUNT(*)                       AS n,
       ROUND(AVG(Revenue), 2)         AS mean_order,
       ROUND(STDDEV_SAMP(Revenue), 2) AS sd
FROM Orders
WHERE Region IN ('North', 'South')
GROUP BY Region;

-- North   5   499.00   235.22
-- South   4   667.50   276.09

5. What 0.05 is, and what it is not

Before the section: you have p = 0.36 and the convention says 0.05. Decide what you would write in a report, in one sentence, before reading mine.

The 0.05 threshold is a convention, chosen for convenience, with no mathematical claim behind it. Nothing changes in the world between p = 0.049 and p = 0.051. Treating the first as a discovery and the second as nothing is the single most common error in applied statistics, and it is an error of habit rather than of arithmetic.

Here are the four things a p-value is regularly claimed to mean, and what it actually means instead. These are worth learning as a set, because each one shows up in real meetings.

What I would actually write for our example: "South's average order was 168.50 higher than North's, on 4 orders against 5. A gap that size arises by chance about a third of the time at these sample sizes, so this data cannot tell us whether the regions really differ. The 95 percent interval on the gap runs from −256 to +593."

That sentence has the effect, the sizes, the test result and the uncertainty, and it never uses the word significant. The interval is doing most of the work, and confidence intervals is where that half of the story lives.

6. Same difference, more rows

Before the table: hold the gap at exactly 168.50 and hold both spreads exactly where they are. Now imagine the same result on bigger groups. Predict what happens to the p-value.

It falls, steeply, and nothing about the finding itself has changed.

This is the fact that makes p-values dangerous when read alone. The same 168.50 gap is "not significant" at 10 orders per region and "highly significant" at 40. The world did not change; only the amount of data did. A p-value is a statement about how much evidence you gathered at least as much as about how big the effect is.

It runs the other way too. Collect a million rows and a difference of 0.3 percent will come back with a p-value of essentially zero, and it will still be a difference nobody should act on. This is why the effect size goes in the report next to the p-value, always, and usually first.

Now picture the last test result you saw quoted in a meeting. Do you know how many rows were behind it? If not, you did not know what the p-value meant, and neither did anyone else in the room.

7. Testing six things at once

Before the arithmetic: you have four regions, so there are six possible pairs to compare. If every region were genuinely identical, and you tested all six pairs at the 0.05 threshold, what is the chance at least one comes back "significant"? Guess before reading on.

Each single test has a 95 percent chance of behaving. Six independent tests all behaving is 0.95 to the sixth power, which is 0.735. So the chance that at least one of the six fires by pure accident is 26.5 percent. Do twenty tests and it is 64 percent.

This is why exploring a data set by running every comparison you can think of, then reporting the ones that came back small, produces findings that evaporate on the next data set. The tests are not wrong individually. The selection afterwards is what breaks them.

Two fixes, and the cheap one is better than most people expect.

Decide the comparison before you look. One planned test at 0.05 is one test. Free, and it is the fix that actually works.

Adjust the threshold if you must run many. The Bonferroni correction divides the threshold by the number of tests: 0.05 ÷ 6 = 0.0083 for our four regions. It is blunt and conservative, and its blunt conservatism is the honest price of having gone fishing.

And if you are exploring rather than testing, say so in the write-up. Exploration is a legitimate and necessary activity. Exploration dressed as confirmation is what produces a chart nobody can reproduce next quarter.

Edge cases that catch people out

Six that each cost somebody a quarter.

Peeking at the p-value as data arrives. Checking a running test every day and stopping when it goes below 0.05 will get you below 0.05 eventually even when nothing is happening, because you are running a new test every day and keeping the best one. Fix the sample size in advance, which is what A/B testing is largely about.

A p-value on the whole population. If your table contains every order that exists, there is nothing to infer. The regions differ by 168.50, full stop. A p-value asks whether a sample generalizes, and there is no sample.

Reporting p without n. Section six is the reason. The two numbers only mean something together.

Treating 0.049 and 0.051 differently. They are the same result. If your decision flips between them, the decision was never resting on the data.

Assuming rows are independent when they are not. Four orders from the same customer are not four independent pieces of evidence, and every test on this page assumes they are. Repeated measures from the same person, sessions from the same account, days from the same store: all of these inflate your apparent sample size and shrink your p-values wrongly.

Confusing a small p with a strong effect. It is worth saying twice. Report the difference, the interval, and the counts. The p-value goes last, and on its own it tells the reader almost nothing.