Prompt research · Maintenance

When do you retire a prompt, and how do you version the panel?

Q: A prompt has returned 0% for two months. Can I drop it?

Only if enough runs sit behind those zeros to make the streak informative. With zero appearances in n runs the 95% upper bound on the true rate is roughly 3/n, so five runs a period over two periods, ten in total, leaves a ceiling near 30%, which is compatible with a prompt you win almost a third of the time. Accumulate around 60 observations per engine before treating a persistent zero as settled. Below that you are not retiring a dead prompt; you are retiring one you never measured.

Q: How do I tell engine drift apart from a real drop?

You cannot tell them apart from one prompt in one period, which is why the panel needs a frozen control subset. A peer-reviewed audit found the same queries two months apart shared only 18% of the pages AI Overviews used, against 45% for organic search, and repeated runs at temperature zero still changed 9 to 28% of decisions. If your control prompts fell in the same period as the ones you acted on, the movement is the engine. If the controls held and the acted-on prompts moved, you have a signal worth reading.

Q: What does "answered without retrieval" actually look like?

It looks like an answer that comes back essentially unchanged when you turn web search off, naming the same companies in the same order. That prompt is reporting the model's stored beliefs rather than anything it just fetched, which means your content cannot move it inside a release cycle. Move it to a small separate panel measured quarterly if the question still matters commercially, and record why it left the main panel. The common mistake is reading such a prompt's flat line as proof that content work does not affect AI visibility.

Q: Can I compare this quarter's number with last quarter's after changing the panel?

Not directly, and the bridge period is what makes the comparison possible. Run both panels in the same week on the same engines and publish both numbers: if the old panel says 34% and the new one says 41% on identical days, seven of those points belong to the instrument and not to the market. Report the change as two series with a stated overlap rather than as one line with a step in it. A single line that steps at exactly the moment you edited the panel is the shape a reader should always distrust.

A panel that quietly gains ten prompts a quarter reports a visibility series whose denominator changed underneath it. That is the most common way this metric gets corrupted, and it never looks like corruption while it is happening.

Topic in Prompt research12 min readUpdated 30 Aug 202610 sources, 8 primary and 2 vendor-described

Back to the roadmap Next: Six things called visibility

On this page

The short answer

Retire a prompt on one of three triggers: it has stopped discriminating at a run count high enough to prove that, it no longer matches a live question, or it is answered without retrieval at all. Replace at the rate you retire and bump the panel version on every change, because the ground moves on its own: a 2026 critical survey records daily source-level Jaccard of 0.34 to 0.42 across four engines over 45 days.¹

Key takeaways

A zero streak proves nothing at low n: with no appearances in n runs the 95% ceiling on the true rate is about 3/n, so ten runs still allow one in three.
Check the engine before you blame the prompt. The same queries two months apart shared only 18% of the pages AI Overviews used, against 45% for organic search.⁵
Bump the version on any change to the set, the wording, the run count or the schedule, then publish one bridge period.
Never add prompts in the period you claim an improvement: a growing denominator moves the rate on its own.

Trigger one

When has a prompt stopped discriminating?

A prompt has stopped discriminating when it returns the same outcome across two consecutive full periods on every engine, and when the run count behind that streak is high enough for the streak to mean something. The second half is the part teams skip, and skipping it retires prompts that were never measured. With zero appearances in n runs, the 95% upper bound on the true rate is roughly 3/n. At 5 runs that ceiling is 45%, at 10 runs 30%, at 30 runs 10% and at 60 runs 5%. A prompt that came back empty ten times is entirely compatible with a real presence rate near one in three.

So set the bar at the bound rather than at the streak: retire on a persistent zero only once the accumulated runs put the ceiling somewhere you genuinely do not care about, which in practice means about 60 observations per engine. The same arithmetic runs upside down, since 60 consecutive appearances put the floor near 95%. Keep two or three saturated prompts in the panel as a tripwire, because if a prompt you always win suddenly drops, something large has happened.

One quieter form of the same failure: a prompt whose presence rate is stable and mid-range while the answer text never changes. That is measuring a fixed retrieval rather than a live competition, and it usually reports a fact about your collection setup rather than the market.

Trigger two

When has a prompt stopped matching a live question?

A prompt has stopped matching a live question when nobody has said anything close to it recently, and the test is a search rather than a judgement: look for the phrasing or a near paraphrase in the last quarter's call recordings and support tickets. If it does not appear, the prompt is measuring your memory of the market rather than the market. Category vocabulary moves quickly, so a panel written eighteen months ago measures a buyer who no longer exists.

A related case is a prompt that was never shaped like a question. Question-phrased queries trigger AI answers far more often than statements: a 55,393-query study measured a Google AI Overview activation rate of 13.7% overall against 64.7% for question-form queries,³ and a representative sample of 11,500 queries accepted at SIGIR 2026 put overall AI Overview presence at 51.5%.⁴ A statement-shaped prompt therefore produces structural zeros that look like a visibility problem and are actually a panel-design problem. Rewrite it as a question, which is a version bump rather than a repair, or retire it.

What is not a retirement trigger: you started ranking first for the query. The domains an engine draws on are not the domains that rank, and the two published audits of the gap disagree on its size while agreeing it is large: on average 53% of the domains AI Overviews consults were absent from the organic top 10 in a 4,706-query audit collected in September 2025,⁵ against 29.8% of reference domains absent from the first page in a 55,393-query study from spring 2026.³ A first-place ranking is not a reason to stop watching what the answer says.

Trigger three

When is a prompt being answered without retrieval?

A prompt is being answered without retrieval when running it with web search disabled returns materially the same answer, naming the same companies in the same order. That prompt is measuring what the model already believes rather than what it just found, and no amount of on-page work moves it inside a release cycle: the lever is years of third-party coverage feeding a future training corpus.

Run the test deliberately rather than opportunistically: same prompt, same day, search off, three runs, on every engine that lets you toggle it. Where the toggle is not exposed, the usable proxy is whether the answer carried any citations at all, logged explicitly as a proxy in the search-enabled field described in the log schema. Retire the prompt from the visibility panel when the test comes back positive, and if the question still matters commercially, move it to a small separate panel measured quarterly.

Retirement here is a reclassification rather than an abandonment, and the reason belongs in the log. It also protects you from reading a flat parametric prompt as evidence that content work does not affect AI visibility, when the question was never answered from the web at all.

The moving floor

Did the prompt move, or did the engine?

A prompt whose result moved may not have moved for any reason you caused, and that background movement is now measured rather than guessed. Summarising Schulte et al. across four engines and 45 days, the 2026 critical survey records daily source-level Jaccard scores of roughly 0.34 to 0.42, with similar levels for repetitions inside 24 hours, and rates engine-to-engine and over-time variation at high confidence.¹ Jaccard is intersection over union rather than a share of cited URLs: 0.34 means about a third of the combined source set from two consecutive days appeared on both. Those numbers come from a small Swiss query universe, so read them as an order of magnitude.

Two other measurements bound the problem. A peer-reviewed audit of 4,706 queries issued in English in September 2025 found the same queries two months apart shared only 18% of the web pages AI Overviews used, against 45% for organic search, and that repeated runs at temperature zero still changed 9 to 28% of decisions.⁵ Across surfaces rather than across time, the representative sample of 11,500 queries put the URL-level Jaccard between Google Search, AI Overviews and Gemini at 0.11 to 0.18, reported in its abstract as below 0.2.⁴ None of that is your content changing.

So no trigger fires on one period of movement. Retire on two consecutive periods, on the accumulated run count, and only after a frozen control subset you never touch shows the same shift missed it.

Versioning

How do you version a panel when you change it?

Bump the version on any change to the set, the wording, the run count or the schedule, stamp it on every logged row, and never compare across versions without a bridge period.

Change	Version bump?	Why
Add or retire a prompt	Yes	The denominator changed, so the rate is a different quantity
Reword an existing prompt	Yes	The wording is the instrument, not a label on it
Change runs per prompt	Yes	The confidence band changes, and comparisons assume the old one
Change the collection schedule	Yes	Batching correlates runs and narrows the apparent interval
Reclassify a prompt's class	Yes	Two class denominators move at once, in opposite directions
Add an engine to the panel	Per engine	Rates are per engine, so old engines keep their series
Fix a typo that changes no words	No	Nothing measurable changed; log it anyway

The bridge period is what turns a version change from a footnote into a number. Run the old panel and the new one in the same week, on the same engines, and publish both: if version 2 reports 34% ±7 and version 3 reports 41% ±7 on identical days, the seven-point gap is the instrument rather than the market. One extra week of collection is cheap next to a quarter of arguing about whether a step change was real.

Keep the frozen control subset inside the panel too, a dozen prompts you never act on and never edit, because their movement is engine drift by construction. That is the baseline you subtract from the prompts you did act on, and it has to absorb a lot: one vendor time series of 230,000 prompts and more than 100 million citations between 14 July and 12 October 2025 recorded one large forum domain's citation rate on a single assistant falling from about 60% of responses to about 10% in six weeks.⁹ That one is vendor-published rather than peer-reviewed, and is cited for direction only.

The quiet corruption

Why does a growing denominator corrupt the number?

A growing denominator corrupts the number because the panel rate is an average over whatever happens to be in the panel, so adding prompts changes the result even when nothing about your visibility has changed. Take a 40-prompt panel reporting 34%, which is 13.6 prompt-equivalents of presence. Add ten easy definitional prompts you already win at 80% and the panel reports 43.2%. Add ten hard comparison prompts you win at 10% instead and it reports 29.2%. Same performance, a 14-point spread, decided entirely by which ten questions somebody added.

It is the most common corruption because adding prompts feels like diligence and nobody records why each one arrived. Three routes are familiar: someone adds prompts after a competitor comes up in a meeting, a sales lead asks for their own question to be tracked, or a content project ships and its target questions get added as a batch. That last one is the worst, because the additions are correlated with the work whose effect you are measuring, which biases the result in the flattering direction by construction.

Four rules remove the problem. Retire and add at the same rate, so panel size holds constant. Bump the version on every change, so the series is explicitly segmented. Publish the class mix beside the rate. And never add prompts in the same period as a claimed improvement: report the old panel's number for that period and start the new one afterwards. Prompt-level noise is already wide enough at realistic run counts that a 2026 statistical treatment of this field concluded many apparent differences between domains sit inside the measurement's own noise floor.²

The honest limit of this article

No published method for panel versioning exists, and none of the thresholds here are validated. The 60-observation bar comes from the rule of three, which is exact only for independent draws, and repeated runs of one prompt are clustered rather than independent, so the true requirement is if anything larger. Retirement is also untestable after the fact: a prompt removed in March might have started discriminating again in June, and because you stopped collecting it there is no way to find out. Retire slowly, keep the retired prompts in a list with the reason attached, and revisit that list once a year.

Where a product fits, and where it does not

All of this is a discipline rather than a feature: a version number, a changelog with a reason per row, and one bridge week per change. No tool enforces it, because retiring a prompt is a judgement about your market that only somebody inside your company can make. Bavior runs a fixed prompt set across five engines on a schedule and records which sources each answer cited, which is what makes the search-off test and the bridge week practical. It does not decide which prompts to retire, cannot tell you whether your category's vocabulary moved, and will not stop you from quietly growing your own denominator. The free AI visibility check and the free GEO audit run without a paid plan; paid plans are from $99/mo billed monthly, or $79.17/mo billed annually (as of 29 Aug 2026).

Sources, all checked 30 Aug 2026

“Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023–2026)”, 15 Jul 2026, arXiv:2607.14035; §6.2 gives daily source-level Jaccard of 0.34–0.42 over four engines and 45 days, summarising Schulte et al. 2026 on a small Swiss query universe. arxiv.org/abs/2607.14035
Sielinski, “Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement”, Mar 2026, arXiv:2603.08924 (preprint). arxiv.org/abs/2603.08924
Xu, Iqbal & Montgomery, 2026 (preprint); 55,393 queries, 13 March to 21 April 2026; AI Overview activation 13.7% overall against 64.7% for question-form queries; 29.8% of reference domains off the first page. arxiv.org/abs/2605.14021
Grossman et al., SIGIR 2026; 11,500 queries; AI Overviews shown for 51.5%; URL-level Jaccard across the three surfaces of 0.11 to 0.18, given in the abstract as <0.2. arxiv.org/abs/2604.27790
Kirsten et al., “Characterizing Web Search in The Age of Generative AI”, Findings of ACL 2026; 4,706 queries, English, September 2025; 53% of consulted domains absent from the organic top 10; 18% page overlap two months apart against 45% organic; 9–28% of decisions flip at temperature zero. aclanthology.org/2026.findings-acl.526
Jaźwińska & Chandrasekar, “AI Search Has a Citation Problem”, Tow Center, Columbia, 6 Mar 2025; 1,600 queries, eight engines. cjr.org
Zhang, He & Yao, “From Citation Selection to Citation Absorption”, arXiv:2604.25707 (preprint); 602 controlled prompts, 21,143 search-layer citations. arxiv.org/abs/2604.25707
Google Search Central, “AI features and your website” (fan-out and eligibility; first-party). developers.google.com/search/docs/appearance/ai-features
Vendor study, 230,000 prompts and 100 million-plus citations, 14 July to 12 October 2025; one large forum domain's citation rate on one assistant fell from about 60% of responses to about 10%. Not linked.
Vendor index, 126 million US AI search prompts, January to April 2026; 36 brands held visibility on every platform measured. Not linked.

Continue reading

Learn GEO GEO roadmap Learn GEO Previous: Ten fields to log Learn GEO Next: Six things called visibility

FAQ

Frequently asked questions.

A prompt has returned 0% for two months. Can I drop it?

Only if enough runs sit behind those zeros to make the streak informative. With zero appearances in n runs the 95% upper bound on the true rate is roughly 3/n, so five runs a period over two periods, ten in total, leaves a ceiling near 30%, which is compatible with a prompt you win almost a third of the time. Accumulate around 60 observations per engine before treating a persistent zero as settled. Below that you are not retiring a dead prompt; you are retiring one you never measured.

How do I tell engine drift apart from a real drop?

You cannot tell them apart from one prompt in one period, which is why the panel needs a frozen control subset. A peer-reviewed audit found the same queries two months apart shared only 18% of the pages AI Overviews used, against 45% for organic search, and repeated runs at temperature zero still changed 9 to 28% of decisions. If your control prompts fell in the same period as the ones you acted on, the movement is the engine. If the controls held and the acted-on prompts moved, you have a signal worth reading.

What does "answered without retrieval" actually look like?

It looks like an answer that comes back essentially unchanged when you turn web search off, naming the same companies in the same order. That prompt is reporting the model's stored beliefs rather than anything it just fetched, which means your content cannot move it inside a release cycle. Move it to a small separate panel measured quarterly if the question still matters commercially, and record why it left the main panel. The common mistake is reading such a prompt's flat line as proof that content work does not affect AI visibility.

Can I compare this quarter's number with last quarter's after changing the panel?

Not directly, and the bridge period is what makes the comparison possible. Run both panels in the same week on the same engines and publish both numbers: if the old panel says 34% and the new one says 41% on identical days, seven of those points belong to the instrument and not to the market. Report the change as two series with a stated overlap rather than as one line with a step in it. A single line that steps at exactly the moment you edited the panel is the shape a reader should always distrust.

Bavior Editorial

The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.

Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com