Citino teamWhy “it went up” is not proof
You add a missing claim to your billing page. Two weeks later your visibility on billing questions is up 4 points. Did the fix work?
Maybe. But three other things could explain it:
- Noise. AI answers vary from run to run. A 4-point move can be chance if your margin of error is ±5.
- The field moved. A model update or a change in how an engine searches can lift every brand on the same days.
- The engines had not read the page yet. If no engine has fetched the changed page, the change cannot have caused anything.
A useful verdict rules out all three.
Step 1: Wait for first retrieval
A fix can only work once an engine has read it. Note when the change went live, then watch for the first time the changed page is cited in an answer, or crawled by the engines’ search bots in your server logs. Until then, the fix is awaiting first retrieval, and no verdict is possible.
Step 2: Measure the questions it targets
Pick, before you look at results, the questions the fix is meant to affect, such as the prompts in your billing segment. Measuring only those keeps the signal from being diluted by unrelated questions, and choosing them in advance stops you from picking the ones that happened to rise.
Measure them daily, with fixed wording, for a fixed window after first retrieval. Thirty days is a sensible default.
Step 3: Subtract the field
Compute your change on those questions over the window, then compute the same change for each competitor named in the same answers, and take the median. Your net effect is your change minus that median.
Worked example (illustrative). After a billing-page rewrite, your visibility on 9 billing prompts rises 9.1 points. The median competitor rose 1.3 points over the same days. Net effect: +7.8 points.
The median is used rather than the average so that one competitor’s unusual week does not swing the result.
Step 4: Compare with the margin of error
Net effect has a margin of error too. Work it out the same way as for visibility itself, by resampling prompts rather than answers (see how to measure AI visibility), and apply it to the difference.
Then the verdict follows:
| Result | Verdict |
|---|---|
| Changed page not yet read by any engine | Awaiting first retrieval |
| Window still running | Measuring |
| Net effect above zero by more than its margin | Confirmed |
| Net effect below zero by more than its margin | Negative |
| Net effect within its margin | Inconclusive |
Report every verdict, including negative and inconclusive ones. A record of fixes that did not work is as useful as one of fixes that did: it tells you which kinds of change to stop making.
Common mistakes
- Judging from the publish date. If engines took ten days to read the page, the first ten days measure nothing.
- Dropping awkward days. Days when every brand moved at once are real data. Label them; do not delete them.
- Choosing questions after the fact. Decide the target questions before looking at the results.
- Reading single answers. “It named us today” is not a verdict.
- Stacking fixes on the same questions. Two changes to the same prompts in the same window cannot be told apart. Stagger them, or judge them together.
How Citino reaches a verdict
Citino records when each fix is published and when an engine first retrieves it, measures the prompts it targets daily, subtracts the median change of your tracked competitors over the same days, and compares the result with a margin of error bootstrapped over prompts. Every fix ends as Confirmed, Negative or Inconclusive, and the full method is on the methodology page.
Everyone tells you what to fix. Citino tells you whether it worked.