Forecast Accuracy: MAE, RMSE, MAPE and the Month That Broke One of Them

1. The error, and the one that cancels itself out

Before the arithmetic: add up the five errors in that last-but-one column, with their signs. Predict roughly what you will get, before reading on.

You get −2,110, which divided by five is −422. That is the bias, or mean error, and it is a genuinely useful number that answers a different question from the rest of this page. It says whether you are systematically forecasting too high or too low. Ours is negative, so the plan ran above reality on average, mostly because of April.

What it cannot do is measure accuracy, because positive and negative errors cancel. A forecast that is 5,000 too high one month and 5,000 too low the next has a bias of zero and is not accurate. Every other measure on this page exists because of that cancellation, and each one deals with it differently: MAE takes the absolute value, RMSE squares.

Report bias alongside accuracy, never instead of it. A forecast with low error and no bias is good. Low error with a consistent bias means you have a correctable offset, which is the cheapest improvement available to any forecast.

2. MAE: the average miss, in your own units

Before the calculation: take the five absolute errors, 230, 155, 185, 2,400 and 30, and average them in your head to the nearest hundred.

(230 + 155 + 185 + 2,400 + 30) ÷ 5 = 3,000 ÷ 5 = 600. The mean absolute error is the average size of a miss, ignoring direction. It is in dollars, so it can be read straight out: this forecast is off by about 600 a month.

Say out loud what is misleading about that sentence, given the table above it. The forecast was off by 230, 155, 185 and 30 in four of the five months, and by 2,400 in one. Nothing was "off by 600" in any month. The average is being dragged by one period, exactly as the mean is dragged by an outlier, and for the same reason.

MAE's advantages are real anyway. It is in the units people care about, every period counts equally, and it is the easiest measure to explain to somebody who does not want a statistics lesson. Its disadvantage is that you cannot compare it across products, regions or time periods with different scales. An MAE of 600 is excellent on a million-dollar line and catastrophic on a two-thousand-dollar one.

import numpy as np
actual   = np.array([2630, 2245, 2585, 0, 2430], float)
forecast = np.array([2400, 2400, 2400, 2400, 2400], float)
error    = actual - forecast

mae  = np.abs(error).mean()          # 600.0
bias = error.mean()                  # -422.0

3. RMSE: the one that hates a single big miss

Before the number: same five errors. Square each, average the squares, take the square root. Predict whether the answer is above or below 600.

Well above: 1,083.70. Squaring makes a 2,400 miss 256 times more expensive than a 150 miss, rather than 16 times, so the one bad month dominates completely.

rmse = np.sqrt((error ** 2).mean())  # 1083.70

The picture at the top of this page is that fact with the numbers removed. Two forecasts, each 400 off in total across four periods. Four misses of 100 give MAE 100 and RMSE 100. One miss of 400 gives MAE 100 and RMSE 200. Identical by one measure, double by the other, and the difference is entirely about how the error was distributed.

Which behaviour is correct depends entirely on what a miss costs you. If being 400 wrong once is much worse than being 100 wrong four times, because it empties a warehouse or blows a budget, then RMSE is measuring what you care about and you should use it. If a miss costs proportionally, so four small misses hurt exactly as much as one big one, MAE is the honest measure and RMSE will make you over-fit to your worst month.

RMSE is always at least as large as MAE, and the gap between them tells you how uneven your errors are. Ours are 600 and 1,083.70, a ratio of 1.8, which is a loud signal that one period is doing most of the damage. When MAE and RMSE are close, your errors are evenly spread.

4. MAPE, and the month that returns infinity

Before the problem: MAPE divides each error by the actual value to get a percentage, then averages those. Look at the April row and say what happens, before reading on.

April's actual is zero. 2,400 divided by 0 is not a number, so the average of the five percentages is not a number either. MAPE is infinite, and depending on your tool you will get inf, a division-by-zero error, or, worst of all, a silently skipped row and a plausible-looking answer.

ape = np.abs(error / actual)
# array([0.0875, 0.0690, 0.0716, inf, 0.0123])
ape.mean()     # inf

This is not an edge case you can dismiss. Any period with no activity gives an actual of zero: a closed store, a product before launch, a channel that went quiet, a week the tracking broke. MAPE is undefined for exactly the periods you most want to understand.

Excluding April, MAPE is 6.01 percent, and the four monthly percentage errors are 8.75, 6.90, 7.16 and 1.23. That number is genuinely informative about the four normal months, and reporting it as "MAPE 6 percent" without saying that a month was dropped is the kind of omission that gets found later.

MAPE has a second, quieter problem: it is asymmetric. Forecasting 200 when the actual is 100 gives an error of 100 percent. Forecasting 100 when the actual is 200 gives 50 percent. The same absolute miss, scored twice as harshly in one direction, which means optimizing for MAPE quietly pushes your forecasts low.

None of that makes MAPE useless. It is unitless, so it compares across products and regions, and non-technical readers understand a percentage immediately. Use it when your actuals are comfortably away from zero, and reach for section five when they are not.

5. WAPE and sMAPE, the two usual repairs

Before the fix: the problem with MAPE is dividing by each individual actual. Say what you would divide by instead, before reading on.

The total. WAPE, the weighted absolute percentage error, adds up all the absolute errors and divides by the sum of the actuals, dividing once instead of five times.

wape = np.abs(error).sum() / actual.sum()
# 3000 / 9890 = 0.3033  ->  30.33%

April no longer breaks anything, because its zero contributes zero to the denominator rather than being a denominator of its own. And the result is naturally weighted by size, so a miss in a big month counts more than a miss in a small one, which is usually what you want. WAPE is my default whenever a percentage is required.

Compare the two on the four normal months, where both work: MAPE 6.01 percent and WAPE 6.07 percent. Almost identical, because those months are all about the same size. The gap only opens when period sizes differ, and then WAPE is the one that reflects the business.

sMAPE, the symmetric version, divides by the average of the actual and the forecast rather than by the actual alone. That fixes the asymmetry and it does not really fix the zero problem: on our data it gives 44.90 percent, which is a strange-looking number driven by April scoring 200 percent, its maximum. sMAPE is common in forecasting competitions and I would not put it in front of a business reader without explaining it, which is usually a sign to use something else.

Read the two columns side by side and the point of this page appears. Removing one month of five turns an average miss of 600 into 150, and turns a positive bias into a negative one. Neither column is dishonest. The one you publish is a decision about whether April is part of the thing you are forecasting, and it belongs in writing next to the number.

6. MASE: better than doing nothing, or not

Before the idea: an MAE of 600 is good or bad compared with what? Name the comparison you would want, before reading on.

The comparison you want is against the effort-free alternative. The standard one is the naive forecast: predict that next month equals this month. Compute its error the same way, then divide.

naive_mae = np.abs(np.diff(actual)).mean()   # 1435.0
mase      = np.abs(error).mean() / naive_mae # 600 / 1435 = 0.418

A MASE below 1 means you beat the naive forecast; above 1 means you did not. Ours is 0.418, so the flat plan of 2,400 was about twice as accurate as guessing that each month repeats the last one. Here are both, scored the same way.

Both beat the in-sample naive baseline, and the flat plan beats it by more. Notice the naive forecast's bias of −14, nearly zero, which is a good illustration of why bias alone proves nothing: a forecast can be unbiased and still be wrong by 1,174 a month, because its errors cancel rather than being small.

MASE is unitless, works when actuals are zero, and is symmetric, which is why forecasting researchers reach for it. Its cost is that it is not intuitive: nobody in a business meeting knows what 0.418 means without the sentence attached. So report it as the sentence. "About twice as accurate as assuming next month repeats this one" is what it says.

Now picture your own forecast, whatever produces it. If you scored it against last period repeated, would it win? That single comparison is worth more than any refinement to the model.

7. Which one to report, and to whom

Before the recommendation: you have six numbers describing one forecast. Decide which you would put on a slide, before reading mine.

My own default block is four numbers: MAE, bias, WAPE and MASE, with the period count and any excluded periods named. It takes one line, it covers the size of the error, its direction, its relative scale and whether the forecast is earning its keep.

The one thing to avoid is picking the measure after seeing the results. If MAPE looks bad and you switch to RMSE without saying so, you have chosen your grade. Decide the measure when you build the forecast, and write it down, for the same reason you decide an A/B test's metric in advance in A/B testing.

Edge cases that catch people out

Six that each cost somebody a quarter.

Zeros in the actuals. The headline case on this page. MAPE is undefined, and several tools will drop the row without telling you, producing a confident number computed on fewer periods than you think.

Scoring on the data you fitted. A model tuned on the same months you then score it against will look excellent and will not be. Hold out periods the model never saw, or score forward as new actuals arrive.

Restated actuals. If last month's actual changes after a data correction, your recorded accuracy changes too. Store the forecast and the actual as they stood when the forecast was scored, with the date, or your accuracy history will quietly rewrite itself.

Comparing MAPE across series with different sizes. A small-volume product will show a terrible MAPE for a tiny absolute miss. That is arithmetic, not performance, and it is why WAPE weighted by size is the fairer comparison.

Aggregating errors before measuring them. Forecast accuracy computed on a total is always better than accuracy computed per product and then combined, because errors cancel in the sum. Both numbers are real, and they answer different questions. Say which one you computed.

One-period-ahead against many. A forecast scored one month ahead is a different task from one scored twelve months ahead, and comparing them is meaningless. State the horizon with the score, always.