A change in AI visibility is real when it is bigger than the margin of error on your numbers and was not shared by every brand in the same answers. If it is smaller than the margin, it is noise, whether it went up or down. The margin has to be calculated over questions, not over single answers, or it will look narrower than it is.
This is the part of AI search analytics that decides whether a chart is telling you something. The rest of this page is the method, with a worked example.
Why the number moves when nothing changed
AI engines do not give the same answer twice. Perplexity says it searches the web in real time, Google says AI Overviews and AI Mode may run multiple related searches to build one response, and ChatGPT searches the web for some questions and not for others. The pages found differ from run to run, and the wording is generated fresh each time.
So your visibility, the share of answers that name you, is an estimate from a sample. Ask the same questions tomorrow and you get a slightly different figure. The question is how different it can be by chance.
The method, in five steps
- Compare two windows of equal length. For example, the last 30 days against the 30 days before. State the change in points (25% minus 22% is 3 points), not as a percentage of a percentage.
- Keep the questions the same on both sides. A question added halfway through belongs to neither window. If it only counts in the later one, the change is partly a change in what you asked.
- Put a margin of error on the number, calculated over questions. Draw your questions at random with replacement, recompute visibility, repeat several hundred times, and take the middle 95% of the results. This is a bootstrap over questions; how to measure AI visibility walks through it.
- Compare the change with the margin. Smaller than the margin: within noise. Bigger: go to step 5.
- Check whether everyone moved. Work out the same change for each competitor named in the same answers and take the median. If the median competitor moved as much as you did, the engine changed, not your standing.
Why the margin must be calculated over questions
Answers to the same question tend to agree from day to day. If an engine names you for “best invoicing software for freelancers” today, it will probably name you tomorrow. Thirty answers to one question are therefore much less than thirty independent pieces of evidence.
The textbook formula for a margin of error on a percentage assumes every answer is independent. Applied to AI answers it gives a margin that is too narrow, and a too-narrow margin turns noise into “changes”.
Worked example (illustrative). You track 40 questions on one engine for 30 days: 1,200 answers. Visibility was 22% in the previous window and is 25% now, a change of +3 points. The textbook formula, treating all 1,200 answers as independent, gives a margin of about ±2.5 points, so +3 looks real. Resampling the 40 questions instead gives, say, ±5.5 points, because a handful of questions decide most of the result. Against ±5.5, a +3 change is within noise. Same data, opposite conclusion. The numbers are fictional.
Reading the result
| What you see (illustrative) | Reading |
|---|---|
| +3 points, margin ±5.5 | Within noise. Report it as steady |
| −4 points, margin ±5.5 | Within noise. Not a loss |
| +9 points, margin ±5.5, median competitor +1 | A real gain, and yours |
| +9 points, margin ±5.5, median competitor +8 | Real, but every brand moved. Not your doing |
| +9 points on one question, 12 answers | Too few answers to judge. Wait |
Noise works in both directions. A drop inside the margin deserves the same shrug as a rise inside it. The discipline is not to explain either one.
What goes wrong, and how to check
| Mistake | How to check |
|---|---|
| Treating every answer as independent | Ask how the margin was calculated. If it shrinks as answers pile up on the same few questions, it is the wrong kind |
| Comparing windows of different lengths | Use the same number of days on both sides |
| Adding or rewording questions mid-window | Compare only questions that ran in both windows. New wording is a new question |
| Reading one question at a time | With 40 questions, a couple will clear their margin by chance alone. Judge groups of questions chosen in advance |
| Pooling engines | A real move on one engine can vanish in the average. Check each engine |
| Dropping odd days | A day when every brand moved is real data. Label it and keep it |
| Judging from a screenshot | One answer is one draw. It cannot show a trend |
One more check when a move does clear its margin: look at what the engine cited before and after. A real change usually has a visible cause, such as a page that started or stopped being read. How to tell whether an AI search fix worked covers the case where the cause is a change you made.
What to ask of an AI visibility tool with a margin of error
If you are choosing a tool to answer this question for you, ask for these:
- A margin on every number, not only on the headline figure.
- Calculated over questions. Ask the vendor directly. “Over answers” means it is too narrow.
- The number of answers behind each figure, so you can see when a figure rests on very little.
- A label for changes inside the margin, and no alerts for them.
- Competitors measured on the same answers, so you can see when everyone moved.
- The method written down where you can read it before you buy.
How often you should check your AI visibility covers the related question of how often to look.
A change smaller than its margin of error is not a small change. It is no evidence of a change at all.
How Citino does this
Citino is an AI search analytics tool. It asks ChatGPT, Google AI Overviews and Perplexity your buyers’ questions every day, with Gemini, Google AI Mode and Claude as add-ons, and keeps every answer.
- Every number carries a 95% margin of error and the number of answers behind it. The margin is a bootstrap that resamples prompts, not answers, as the methodology sets out. The resampling uses a fixed seed, so the same window always gives the same margin.
- A change is the current window minus the previous window of equal length, in points. A prompt that was not running in the previous window is left out of both sides.
- A change smaller than its margin is shown grey, labelled within noise, and sends no alert.
- A figure that rests on few answers is still shown, with its margin and a low-sample badge beside it. Smaller prompt sets have wider margins, and the margin says so.
- Days when most brands moved at once on one engine are labelled and kept, never removed from the average.
The margin of error is on every plan; plans differ only in the number of prompts. See pricing and how Citino measures.