Is my AI visibility change real or just noise?

The short answer

A change in AI visibility is real when it is bigger than the margin of error on your numbers and was not shared by every brand in the same answers. Calculate the margin over questions, not over single answers, compare two windows of equal length, and treat anything inside the margin as noise, whichever way it points.

Updated 8 min read

A change in AI visibility is real when it is bigger than the margin of error on your numbers and was not shared by every brand in the same answers. If it is smaller than the margin, it is noise, whether it went up or down. The margin has to be calculated over questions, not over single answers, or it will look narrower than it is.

This is the part of AI search analytics that decides whether a chart is telling you something. The rest of this page is the method, with a worked example.

Why the number moves when nothing changed

AI engines do not give the same answer twice. Perplexity says it searches the web in real time, Google says AI Overviews and AI Mode may run multiple related searches to build one response, and ChatGPT searches the web for some questions and not for others. The pages found differ from run to run, and the wording is generated fresh each time.

So your visibility, the share of answers that name you, is an estimate from a sample. Ask the same questions tomorrow and you get a slightly different figure. The question is how different it can be by chance.

The method, in five steps

  1. Compare two windows of equal length. For example, the last 30 days against the 30 days before. State the change in points (25% minus 22% is 3 points), not as a percentage of a percentage.
  2. Keep the questions the same on both sides. A question added halfway through belongs to neither window. If it only counts in the later one, the change is partly a change in what you asked.
  3. Put a margin of error on the number, calculated over questions. Draw your questions at random with replacement, recompute visibility, repeat several hundred times, and take the middle 95% of the results. This is a bootstrap over questions; how to measure AI visibility walks through it.
  4. Compare the change with the margin. Smaller than the margin: within noise. Bigger: go to step 5.
  5. Check whether everyone moved. Work out the same change for each competitor named in the same answers and take the median. If the median competitor moved as much as you did, the engine changed, not your standing.

Why the margin must be calculated over questions

Answers to the same question tend to agree from day to day. If an engine names you for “best invoicing software for freelancers” today, it will probably name you tomorrow. Thirty answers to one question are therefore much less than thirty independent pieces of evidence.

The textbook formula for a margin of error on a percentage assumes every answer is independent. Applied to AI answers it gives a margin that is too narrow, and a too-narrow margin turns noise into “changes”.

Worked example (illustrative). You track 40 questions on one engine for 30 days: 1,200 answers. Visibility was 22% in the previous window and is 25% now, a change of +3 points. The textbook formula, treating all 1,200 answers as independent, gives a margin of about ±2.5 points, so +3 looks real. Resampling the 40 questions instead gives, say, ±5.5 points, because a handful of questions decide most of the result. Against ±5.5, a +3 change is within noise. Same data, opposite conclusion. The numbers are fictional.

Reading the result

What you see (illustrative)Reading
+3 points, margin ±5.5Within noise. Report it as steady
−4 points, margin ±5.5Within noise. Not a loss
+9 points, margin ±5.5, median competitor +1A real gain, and yours
+9 points, margin ±5.5, median competitor +8Real, but every brand moved. Not your doing
+9 points on one question, 12 answersToo few answers to judge. Wait

Noise works in both directions. A drop inside the margin deserves the same shrug as a rise inside it. The discipline is not to explain either one.

What goes wrong, and how to check

MistakeHow to check
Treating every answer as independentAsk how the margin was calculated. If it shrinks as answers pile up on the same few questions, it is the wrong kind
Comparing windows of different lengthsUse the same number of days on both sides
Adding or rewording questions mid-windowCompare only questions that ran in both windows. New wording is a new question
Reading one question at a timeWith 40 questions, a couple will clear their margin by chance alone. Judge groups of questions chosen in advance
Pooling enginesA real move on one engine can vanish in the average. Check each engine
Dropping odd daysA day when every brand moved is real data. Label it and keep it
Judging from a screenshotOne answer is one draw. It cannot show a trend

One more check when a move does clear its margin: look at what the engine cited before and after. A real change usually has a visible cause, such as a page that started or stopped being read. How to tell whether an AI search fix worked covers the case where the cause is a change you made.

What to ask of an AI visibility tool with a margin of error

If you are choosing a tool to answer this question for you, ask for these:

  • A margin on every number, not only on the headline figure.
  • Calculated over questions. Ask the vendor directly. “Over answers” means it is too narrow.
  • The number of answers behind each figure, so you can see when a figure rests on very little.
  • A label for changes inside the margin, and no alerts for them.
  • Competitors measured on the same answers, so you can see when everyone moved.
  • The method written down where you can read it before you buy.

How often you should check your AI visibility covers the related question of how often to look.

A change smaller than its margin of error is not a small change. It is no evidence of a change at all.

How Citino does this

Citino is an AI search analytics tool. It asks ChatGPT, Google AI Overviews and Perplexity your buyers’ questions every day, with Gemini, Google AI Mode and Claude as add-ons, and keeps every answer.

  • Every number carries a 95% margin of error and the number of answers behind it. The margin is a bootstrap that resamples prompts, not answers, as the methodology sets out. The resampling uses a fixed seed, so the same window always gives the same margin.
  • A change is the current window minus the previous window of equal length, in points. A prompt that was not running in the previous window is left out of both sides.
  • A change smaller than its margin is shown grey, labelled within noise, and sends no alert.
  • A figure that rests on few answers is still shown, with its margin and a low-sample badge beside it. Smaller prompt sets have wider margins, and the margin says so.
  • Days when most brands moved at once on one engine are labelled and kept, never removed from the average.

The margin of error is on every plan; plans differ only in the number of prompts. See pricing and how Citino measures.

Questions people ask

Is my AI visibility change real or just noise?

It is real if the change is bigger than the margin of error on the number and your competitors did not move by the same amount over the same days. If the change is smaller than the margin, treat it as noise. Calculate the margin over questions, not over single answers, or it will look narrower than it is.

Which AI visibility tools show a margin of error?

Ask any tool three things: does every number carry a margin of error, is the margin calculated over questions rather than answers, and does the tool label changes inside it as noise. Citino shows a 95% margin of error on every number, calculated by resampling prompts, and labels a change inside it within noise.

How big does a change in AI visibility need to be to count?

Bigger than its margin of error. There is no fixed number of points, because the margin depends on how many questions you track, how many answers you have and how much the answers vary. A set of 20 questions has a wider margin than a set of 200.

Why does my AI visibility change from day to day when I changed nothing?

AI engines do not give the same answer twice. Many answers are built from a live web search, the pages found differ from run to run, and the wording is generated fresh each time. Daily figures move for these reasons alone, which is why a single day or a single answer is not evidence.

See where AI names you
on every engine, every day

Citino asks the questions your buyers ask and tells you whether your fixes worked.

Experiments · net of competitors
  • Crateline comparison page+4.1 ptsConfirmed
  • 3PL billing page rewrite+7.8 so farMeasuring
  • Shopify integration guide−4.5 ptsNegative
  • Pricing page refreshwithin noiseInconclusive