Real change or random noise? A plain guide to reading your numbers

Every number you track moves from one period to the next, even when nothing has changed. Weekly sales, daily sign-ups and monthly churn all wobble. The skill is separating a move that is bigger than the wobble from one that is not.

Compare the move to the usual wobble

Look at how much the number normally changes between neighbouring periods. A drop that is about the same size as ordinary ups and downs tells you nothing. A drop several times larger than anything in the history is worth a closer look. Software does this with a statistical test; by hand, scan the history and ask "has it ever moved this much before?"

Mistake 1: small groups

Three orders last month and six this month is a doubling, and it is almost certainly noise. The fewer records behind a number, the more it jumps around. Always ask how many records sit behind a percentage.

Mistake 2: looking at too many things

If you examine a hundred segments, a few will look unusual by luck alone. The more comparisons you make, the stricter your bar should be for calling one of them real. A finding that survives a strict bar after you searched widely is far more believable than one you went looking for.

Mistake 3: ignoring the calendar

A December spike that happens every December is a pattern, not news. A weekend dip in a business that sells to offices is the same. Check whether the "change" appears in the same place in earlier years before you call it new.

Mistake 4: letting one big customer decide the average

One very large order can lift an average and make a whole month look strong. Recalculate without it, or look at the median. If the story disappears, it was one customer, not a trend. The same goes for a group made of just a handful of customers: their many orders are not many independent observations.

Mistake 5: reading cause into a link

Two numbers that rise and fall together are linked, which is a good reason to investigate and not a reason to conclude. Ad spend and sales may both rise in the same busy season. A finding tells you where to look; it does not tell you why.

How AnalyzeIt guards against these

Discovery compares each move with the series’ own history, applies a stricter bar when it has searched many places, checks for yearly seasonality when there are at least two full years of data, treats customer-clustered groups as the few independent customers they are, and labels each finding by the strength of its evidence: Confirmed in the data, Strong evidence, Likely, or Worth checking. It tested itself on realistic data with no real pattern before we trusted it, and on data with a planted pattern to see how often it was found.

Read more: How discovery works, or try the live demo with no account.