Home/Learn GEO/Why answers change

How AI search works · Topic

Why AI answers change between two identical runs of the same question.

Repeat the same prompt on the same engine an hour later and you get a different set of sources. That is not a glitch, it is the measured behaviour of the system, and it decides what counts as evidence about your own visibility.

Topic in How AI search works12 min readUpdated 30 Aug 20267 sources: 6 primary, 1 vendor-described

Back to the roadmap Next: Do AI engines agree

On this page

The short answer

AI answers change between identical runs because several stages of the pipeline are sampled rather than computed, and the index underneath them keeps moving. Temperature governs only the last of those stages, so setting it to zero does not make the system reproducible. Across four engines and 45 days, Schulte et al. measured daily source-level Jaccard scores of 0.34–0.42 for the sources one query returns, and a peer-reviewed audit of 4,706 queries found 9–28% of decisions changing on repeated runs at temperature zero, both as reported in the 2026 critical survey.¹

Key takeaways

Four things vary between two runs: query fan-out, a live index, reranking at the context cut line, and generation. Temperature touches only the last one.
Jaccard is intersection over union, not the share of cited URLs that carried over. At the measured 0.34–0.42, a repeat run reliably returns sources the first run never showed you.
One screenshot is one draw. Report a share per engine with the run count beside it, because engines disagree with each other at Jaccard 0.11–0.18 even inside one company.

Mechanism

What exactly is random inside an AI answer engine?

At least four separate things vary between two runs of the same prompt, and only one of them is the language model's sampling temperature. Working through the pipeline described in the how AI search works stage, the first is query expansion: Google documents that AI Overviews and AI Mode “may use a ‘query fan-out’ technique … to develop a response”, issuing multiple related searches across subtopics and data sources.³ Those sub-queries are generated, not looked up, so a different set of sub-queries produces a different candidate pool before any ranking happens.

The second is retrieval itself. Each generated sub-query hits a live index that is re-crawled continuously, sharded across machines, and subject to freshness and personalisation logic. The third is selection: candidates are merged, deduplicated, reranked and cut to a context budget, and near-ties at the cut line resolve differently on different runs. The fourth is generation, where the model decides which of the selected sources to actually attach to which sentence: a source can be retrieved, read, and then not cited.

Two of those four sit upstream of anything you control, which is why turning temperature to zero does not make the system deterministic. The 2026 critical survey states it plainly: a reported temperature of zero “fixes neither the index, nor retrieval, nor the versions of external services.”¹

Magnitude

How different are two identical runs, in numbers?

Overlap between one day's sources and the next sits at a Jaccard of 0.34–0.42, measured by Schulte et al. across four engines over 45 days on a small universe of Swiss queries.

Jaccard similarity is the intersection of two sets divided by their union: identical source sets score 1.0, sets with nothing in common score 0. It is not the share of sources that repeated.

0.34–0.42

Overlap between repeat runs

Daily source-level Jaccard for the sources one query returns on commercial engines, across four engines and 45 days, reported in the 2026 critical survey.

9–28%

Repeated-decision changes

Share of repeated decisions that flip at temperature zero, in a peer-reviewed audit of 4,706 queries. Determinism in the generation step does not buy determinism in the answer.

power law

Shape of the citation distribution

Citation counts across domains follow a power law with substantial variability. A handful of domains are cited constantly; a very long tail is cited unpredictably.

A Jaccard of 0.38 is easier to feel as an example. If an engine cites eight sources per answer and two runs overlap at 0.38, the union across both runs is about eleven distinct sources and roughly five of the eight repeat. Run it a third time and you will meet sources you have not seen yet. Nothing about your page changed; the sample did.

The March 2026 statistical framework adds the finding most people skip: rankings inside the frequently-cited set are unstable, not merely at the top but throughout, and “many apparent differences between domains fall within the noise floor of the measurement process.”² That is why so much AI-visibility reporting is confidently wrong: a real number measured once, with no interval attached, cannot be told apart from the measurement's own jitter.

The noise floor

Why do most brands sit inside the noise instead of above it?

Because a power-law distribution puts almost everyone in the flat tail, where run-to-run variance is larger than the difference between neighbours. If a domain appears in 6% of answers for a prompt set and a competitor appears in 9%, that three-point gap is smaller than the spread you would see by re-running the same prompt set on a different afternoon. That is what the March 2026 bootstrap intervals show on real domains: gaps of that size sit inside the noise floor.²

The arithmetic of why is not controversial and does not need a study. For a share measured on n independent runs, the half-width of the 95% Wilson interval around a value near 50% is roughly 1.96 divided by twice the square root of (n plus 3.84). Five runs gives about ±33 points, ten runs about ±26, thirty runs about ±17, one hundred runs about ±9.6. Comparing two such measurements, this month against last or you against a competitor, needs the two intervals not to overlap, which costs roughly another 40% in sample size. That is binomial arithmetic, not a finding from any paper, and it is why a five-run check cannot detect anything smaller than an enormous change.

That yields a hierarchy of what is detectable on a real budget. “We appear in some answers and not others” takes a handful of runs; “we went from never appearing to routinely appearing” takes a few dozen; “we gained three points of share against a competitor” is not detectable at any sample size a small team will actually run.

Drift

Does the answer also change because the web changed?

Yes, and that drift is separate from run-to-run randomness, larger, and slower, which makes it easy to mistake for a result. Three distinct clocks run under any measurement you take. The index changes as pages are published, updated and removed. The engine changes as models and retrieval stacks are swapped, usually without announcement. And the crawling layer changes: Cloudflare's 2025 year-in-review reports AI bots at 4.2% of HTML requests across the year, with user-triggered crawling growing more than fifteen-fold and one assistant's user-triggered agent peaking at sixteen times its start-of-year volume.⁴

The sharpest published illustration of engine drift comes from a commercial three-month study of 230,000 prompts and over 100 million citations, run between mid-July and mid-October 2025, which recorded one assistant's citation rate for a single large discussion platform falling from roughly 60% of responses to roughly 10% in about six weeks, with an encyclopedia source dropping by a similar order in the same window.⁵ No publisher did anything to cause that. A team that ran a before-and-after across that window would have measured an enormous effect from its own content work and been wrong about the cause.

A fourth, quieter source of change is source availability: a 2026 preprint, reported via the critical survey, found 27.1% of URLs already cited in AI answers could not be scraped, being inaccessible, removed or non-textual.⁶ A source that disappears behind a login between two of your runs leaves the answer without anything on your side changing.

The honest limit of this article

The 0.34–0.42 and 9–28% figures reach you through a survey aggregating studies across engines and time periods, not from one controlled experiment on the products you use, and the Jaccard range comes from a small universe of Swiss queries. No engine publishes its own run-to-run variance. The plus-or-minus-1-over-root-n arithmetic assumes independent draws, which repeat runs inside one conversation are not: session state and caching correlate them, so an interval from same-session runs is optimistic. The 27.1% figure measures whether a third-party researcher could fetch a cited URL, not whether the engine that cited it could. Treat every number here as an order of magnitude, not a constant.

Evidence quality

Why is a screenshot of an AI answer not evidence?

A screenshot is a single draw from a distribution whose spread is wider than almost any effect a content team is trying to detect. It is still useful for three narrow things: proving an answer of a given shape is possible, capturing a specific factual error for a correction request, and showing a stakeholder what the surface looks like. It cannot support a claim about frequency, ranking, change over time, or a competitor.

There is a second failure mode screenshots hide. Engines differ from each other more than they differ from themselves: the 11,500-query study presented at SIGIR 2026 reports average Jaccard similarity below 0.2 between the sources returned by Google organic results, AI Overviews and Gemini, with the pairwise figures running 0.11 to 0.18, across three surfaces from one company.⁷ A screenshot of one surface says nothing about the others, so “we checked and we are not there” is usually a statement about one surface on one afternoon. Which surface, which mode, which locale, and whether search was enabled all change the answer, and none of that is visible in the image.

Method

How many runs do you need before a number means anything?

The 2026 survey's own starting point is seven to eight repetitions per prompt, repeated until the interval around the estimate is narrow enough for the decision at hand.¹ One run is a sample of one, drawn from a distribution wider than most effects you are trying to see.

The procedure below runs in a spreadsheet.

Fix the prompt set in writing before you look at a single answer

Twenty to fifty prompts a real buyer would type, written down and frozen. Adding a prompt after you see a good answer is how a measurement becomes a marketing asset. Keep the prompts that return nothing about you: they are the denominator.

Record the conditions, every time

Product, mode, model version, date, locale, and whether search was enabled. The 2026 survey lists exactly this as the minimum any study in this field must report, because a result without those fields cannot be compared to anything, including your own earlier run.

Spread the runs across days, not tabs

Thirty runs in one hour largely re-sample one index state. Thirty runs across three weeks sample the thing you actually care about. Where you can, use a fresh session per run so the draws are closer to independent.

Report a share with an interval, per engine

Cited in 11 of 40 runs on one engine is a sentence you can defend. A single blended visibility score across engines is not, because the engines disagree at Jaccard 0.11–0.18 even inside one company.

Log the cited sources, not just whether you appeared

The list of URLs an answer cited is stabler and more actionable than the prose, and it tells you which third-party pages you are actually competing against. Most of them will not be yours.

Change one thing, then wait longer than feels reasonable

Recrawl and reindex delays mean a page edit and its effect are weeks apart, and engine-side drift over those weeks can be larger than your edit. Two changes shipped in one week cannot be told apart afterwards.

The output is a range, which is less satisfying than a rank and considerably more useful. The measuring AI visibility stage turns it into a full measurement plan. The habit worth building today is smaller: whenever you write down an AI-visibility number, write beside it how many runs it came from. A number without an n is not a measurement.

Where a product fits, and where it does not

Everything above works with a spreadsheet, a calendar reminder and enough patience to run the same prompts for a month. What Bavior does for this specific problem is remove the patience requirement: it runs a fixed prompt set against five engines on a schedule and records which sources each answer cited, so what you read is a distribution across many runs rather than one afternoon's screenshot. It does not reduce the variance, cannot explain why any individual answer changed, and will never tell you that a three-point move is real when the sample says it is noise. The free AI visibility check and free GEO audit run without a paid plan; paid tracking starts from $99/mo billed monthly, or $79.17/mo billed annually (as of 29 Aug 2026).

Sources, all checked 29 Aug 2026

Martinez, “Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023–2026)”, 15 Jul 2026, arXiv:2607.14035 (survey preprint). §6.2: Schulte et al. daily Jaccard 0.34–0.42 (four engines, 45 days, small Swiss query universe); Kirsten et al. peer-reviewed audit, 4,706 queries, 9–28% repeated-decision changes at temperature zero: arxiv.org/abs/2607.14035
Sielinski, “Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement”, Mar 2026, arXiv:2603.08924 (preprint; three platforms, nine daily collections plus ten-minute sampling): arxiv.org/abs/2603.08924
Google Search Central, “AI features and your website” (query fan-out; first-party documentation, last updated 10 Dec 2025): developers.google.com/search/docs/appearance/ai-features
Cloudflare, “Radar 2025 Year in Review” (data 1 Jan – 2 Dec 2025; AI bots 4.2% of HTML requests, user-action crawling up more than 15×, one agent peaking at 16×): blog.cloudflare.com/radar-2025-year-in-review
Three-month citation time series, 230,000 prompts and 100M+ citations across three assistants, 14 Jul – 12 Oct 2025; the discussion-platform collapse described above. Vendor-published; described rather than linked, per this curriculum's sourcing rule.
Allaham & Diakopoulos, 2026 preprint (reported in arXiv:2607.14035 §8.3): 27.1% of URLs already cited in AI answers could not be scraped by the researchers because they were inaccessible, removed or non-textual
Grossman et al., SIGIR 2026, 11,500 queries; average Jaccard below 0.2 in the abstract, pairwise 0.11–0.18 in the contributions list, across Google organic, AI Overviews and Gemini: arxiv.org/abs/2604.27790

Continue reading

Learn GEO GEO roadmap Learn GEO Previous: Where GEO advice goes wrong Learn GEO Next: Do AI engines agree

FAQ

Frequently asked questions.

Does setting temperature to zero make AI answers reproducible?

No. Temperature controls sampling inside the generation step only, and generation is the last of at least four stages that vary. Query fan-out generates a different set of sub-queries, retrieval hits a live index that changes continuously, and reranking resolves near-ties differently between runs. A peer-reviewed audit of 4,706 queries, reported in the 2026 critical survey, found 9–28% of decisions changing on repeated runs at temperature zero, on exactly the surfaces where temperature can be held fixed.

How many runs do I need to detect that my content work moved the needle?

More than most teams run, and the honest answer depends on the size of the effect. For a share measured near 50%, the half-width of the 95% Wilson interval is roughly 1.96 divided by twice the square root of (the number of runs plus 3.84): ten runs gives about ±26 points, thirty gives ±17, one hundred gives ±9.6. Comparing before against after needs both intervals to clear each other, which costs roughly another 40% in sample size. That is ordinary binomial arithmetic, not a study finding, and it means a five-run spot check can only detect changes that are obvious without measuring.

If the answer changes every time, is measuring AI visibility worth doing at all?

Yes, but as a distribution rather than a rank. "Cited in 11 of 40 runs on this engine in August" is a defensible sentence that supports decisions; "we rank third in ChatGPT" is not a thing that exists. The useful outputs of a sampling programme are the share of runs you appear in per engine, the list of third-party sources that keep getting cited instead of you, and whether both move over a quarter. All three survive the variance; a single ranking claim does not.

Why did my AI visibility drop without me changing anything?

Most likely because the engine changed, not because you did. A commercial three-month study of 230,000 prompts recorded one assistant's citation rate for a single large discussion platform collapsing from roughly 60% of responses to roughly 10% in about six weeks during 2025, with no action by any publisher involved. Source availability shifts too: a 2026 preprint, reported via the critical survey, found 27.1% of URLs already cited in AI answers could not be scraped, a figure that measures whether a third-party researcher could fetch them, not whether the citing engine could. Before diagnosing your own content, check whether the whole source mix for those prompts moved.

Is variance the same across every AI engine?

No, and the engines also disagree with each other far more than they disagree with themselves. An 11,500-query study presented at SIGIR 2026 reports average Jaccard similarity below 0.2 between Google organic results, AI Overviews and Gemini, with pairwise figures of 0.11 to 0.18, against day-to-day overlap of 0.34–0.42 within a single surface. Cross-engine disagreement is the larger effect, which is why results have to be reported per engine and per surface rather than averaged into one number.

Bavior Editorial

The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.

Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com