Home/Learn GEO/Checklist
Technical GEO · Synthesis

The technical GEO checklist, in the order you should run it.

Ten checks in three tiers. Each states what to run, what a pass and a fail look like, and which article to read when it fails. The order is not cosmetic: the items are not independent.

On this page
Share this
Share on X Share on LinkedIn
The short answer

The technical checklist is ten checks in three tiers, run cheapest first: four a single plain fetch settles, three only your own server logs can answer, and three that are decisions rather than fixes. The order carries the argument, because the items are not independent: a failure near the top makes every measurement below it unreadable. Among URLs that AI answers had already cited, 27.1% could not be fetched at all, being inaccessible, removed or non-textual, which is the tier-one failure rate among pages the engines had already chosen.8

Key takeaways
  • Tier one is one fetch per important URL: status code, body text in the raw response, robots result per search-side token, snippet directives. Nothing below it can be trusted until those four pass.
  • Tier two exists because tier one tests what your server is configured to do, while the logs record what it did. A bot-management rule outranks robots.txt and does not announce itself.
  • Tier three is not a fix. Gates, user-triggered fetchers and last-modified honesty are policy calls, and the right answer for a news site is the wrong one for documentation.
  • Passing everything buys eligibility, not citation. Two vendors document about 24 hours before a robots.txt change is reflected, so re-check the next day.
The order

Why is the order of this checklist the whole checklist?

A checklist is worth something only when its items are ordered by dependency, and technical items are unusually dependent on each other. Four of the ten below are settled by one command-line fetch. Three need a month of retained server logs and somebody who can query them. Three are not fixes at all: they are decisions about what you want other people’s machines to be able to read. Running them in that sequence is not tidiness; it is the only sequence in which the results mean anything.

The dependency runs one way. If a plain fetch returns a challenge instead of a page, every log you examine afterwards shows silence, for a reason that has nothing to do with an engine’s interest in you. If robots.txt disallows the search-side token, a month of crawl data reads as indifference, and a team concludes that AI engines ignore its category when the site told them not to come. If the body text only exists after hydration, a citation measurement measures the JavaScript bundle rather than the writing. Each is a false negative that looks exactly like a true one, which is why a failure high on the list is a stop instruction and not a note to carry forward.

Cost points the same way. Tier one is a laptop and ten minutes. Tier two costs a person a day and needs log retention you may not have. Tier three costs a meeting. Cheap checks gate expensive ones, so the response to a tier-one failure is to fix it, wait for the vendor systems to catch up, and start again from the top. The reasoning behind each item lives in its own article, linked from the item; this page adds the sequence and the run instructions.

Tier one

What can one plain fetch settle in ten minutes?

Four checks, one command each, no tooling and no account. Run them against every URL you would want quoted, not the homepage alone.

01

Status code, under each token

Fetch each URL under each search-side user agent: curl -sIL -A OAI-SearchBot https://example.com/page. Pass is a 200 with at most one redirect hop. Fail is a 403, 429, 503, an interstitial, or a chain. When it fails, read paywalls and crawler blocks.

02

Body text in the raw response

Save the raw response to a file and look in it for a sentence you would want quoted. Pass is finding that sentence in the bytes the server sent. Fail is finding it only once the page has run in a browser. When it fails, read whether AI crawlers run JavaScript.

03

robots.txt result, token by token

Evaluate the effective result separately for Googlebot, OAI-SearchBot,3 Claude-SearchBot,4 PerplexityBot5 and Applebot,6 then confirm each with a real fetch. Pass is every search-side token allowed; fail is usually an inherited wildcard disallow. When it fails, read which crawler tokens matter.

04

Indexed and snippet-eligible

Google’s stated requirement is that the page “must be indexed and eligible to be shown in Google Search with a snippet.”1 Pass is a self-referential canonical with no noindex, nosnippet or max-snippet directive in the head or the headers. When it fails, read crawlability and indexation.

Record results per URL, not per site. A homepage almost always passes all four; the deep pages carrying your actual answers are where tier one fails, and those are the pages an engine needs. Two of the four also fail together often, because a client-rendered application tends to sit behind an edge configuration with a bot rule attached. Why this caps the rest of the work is argued in retrievability as a ceiling.

Tier two

What can only your own logs answer?

Tier one tests what your server is supposed to do. The logs record what it actually did, and the two disagree more often than anyone expects.

05

Arrivals, per token per day

Filter thirty days of access logs by user agent, counting each search-side token separately. Pass is a non-zero, roughly steady count for every token you allowed. Fail is zero for a token robots.txt permits: something between the file and the log is dropping them. Then read what AI crawl traffic looks like.

06

Wasted fetches

Count the share of each token’s requests returning a 404 or a redirect. One infrastructure provider’s December 2024 log study reported 34.82% of ChatGPT’s crawler fetches and 34.16% of Claude’s landing on 404s, against 8.22% for Googlebot.7 Pass is sitting nearer Googlebot; a gap points at a stale sitemap or retired URLs.

07

Silent challenges

Look for 403, 429 and 503 responses by token, reading the firewall’s own event log rather than the web server’s alone. Perplexity documents that owners may need to allowlist its bots in a web application firewall explicitly.5 Pass is no challenge events against an allowed token.

Item 07 is the one that most often contradicts the paperwork. A robots directive is a request a well-behaved crawler honours; a bot-management rule is an enforcement that fires before your application sees anything, and it is usually owned by a different team. When the two disagree the enforcement wins, and the robots file stays wrong for months without producing a symptom anyone would look at. A team that ran a before-and-after across a robots.txt correction, and saw crawl volume refuse to move, found the cause in a managed bot ruleset at the edge rather than the file it had just edited. How to read that data, and how to tell a real crawler from something wearing its name, is covered in reading crawler logs.

Tier three

What is left that is a decision rather than a fix?

The last three items have no correct answer a checklist can supply. They need somebody who can say what the site is for.

08

Gates, on purpose or by accident

Decide explicitly which content sits behind a login, a paywall or a consent wall, and write the decision down. Pass is a gate somebody chose and owns. Fail is a gate nobody chose, usually a consent interstitial from a tag manager. The trade is argued in paywalls and crawler blocks.

09

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User fetch because a person asked, not on a crawl schedule, and Perplexity documents that “since a user requested the fetch, this fetcher generally ignores robots.txt rules.”5 Pass is having decided how to treat them. Treating them like crawlers is a choice, not a default.

10

A last-modified date that is true

Compare the Last-Modified header and any published modification date against the last real edit. Pass is that they agree. Fail is a build stamping today onto every file, which makes freshness meaningless across the whole domain rather than one page.

None of the three has a pass a script can compute, which is why they sit at the bottom. They also change least often: item 08 when the business model changes, item 09 when a vendor publishes new documentation, and item 10 once, when somebody fixes the build. Everything above them deserves a schedule, because what breaks those items is deployments and infrastructure changes nobody labelled as either.

Scope

Which items are deliberately not on this list?

On the list, in order

  • Status code per search-side token
  • Body text in the raw response
  • robots.txt result per token
  • Indexed, canonical clean, snippet allowed
  • Arrivals per token in your logs
  • 404 and redirect share per token
  • No silent firewall challenge
  • Gates deliberate and owned
  • A policy for user-triggered fetchers
  • Last-modified dates that are true

Not on the list, and why

  • llms.txt: Google’s guidance says Search does not use such files
  • Schema added hoping to win citations, rather than for rich results
  • Blocking a training token, which changes what a model learns, not what an answer cites
  • Blanket wildcard disallows, which remove the search crawlers you wanted
  • Any tactic sold on crawler hit-volume with no stated sample

The exclusions are not a matter of taste. Google’s guidance states that you “don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.”2 That sentence is first-party and dated, which is more than the tactics it rules out can offer. Structured data stays worth shipping for rich results, where the evidence is old and good; it is the citation claim attached to it that has nothing behind it. A list holding every plausible-sounding item also defeats its own purpose: ten checks get run, twenty-five get skimmed, and the four that mattered sit in the skimmed part.

Cadence

How often should you re-run it?

Re-run tier one after every deployment that touches routing, rendering, response headers or the edge configuration, which in most teams means weekly rather than quarterly. Re-run tier two monthly, because its signal is a trend and one day of logs is noise. Tier three moves when the business changes rather than when the code does.

Wait a day before believing a re-check. OpenAI’s documentation says it “can take ~24 hours from a site’s robots.txt update for our systems to adjust,”3 and Perplexity’s says a change “may take up to 24 hours” to be reflected.5 Re-testing an hour after a fix and finding the old behaviour is the most common way a team talks itself out of a correct change.

Keep the output somewhere durable, one row per URL per run, with the date on it. The value of the second run is its diff against the first, and a checklist living in a terminal history produces no diff. Two columns are worth carrying beyond pass and fail: which item failed, and what shipped between the runs. That pairing is the only causal evidence this exercise can generate; it is weak, and it is the kind you can actually collect.

The honest limit of this checklist

A list of ten items implies that the ten are exhaustive and that a full pass is a result. Neither is true. It is exhaustive only of failures cheap to detect from outside the site and from your own logs; a page can pass all ten and still match none of the queries an engine issued, which is a coverage problem this page cannot see. Passing establishes eligibility, not citation, and between the two sit every ranking, selection and synthesis decision made afterwards. The 2026 critical survey rates the claim that citation scores predict clicks, conversions or revenue at very low confidence.9 Read a completed checklist as an obstacle removed, never as an outcome: tier one rests on first-party vendor documentation and is strong, while evidence connecting any of it to money does not exist yet.

Where a product fits, and where it does not

Every item above is free. Tier one needs curl and ten minutes, tier two needs log access you already have, and tier three needs a decision no software can make for you. Bavior does none of it: it does not crawl your site, does not read your robots.txt, and cannot make a blocked page fetchable. It covers the stage after this list, running a fixed prompt set across engines on a schedule and recording which sources each answer cited, which is how you learn whether a tier-one repair changed who gets quoted. Run the checklist first: measuring an unretrievable page measures nothing. How much of an answer you can own is taken up in how much of the answer you own.

Sources, all checked 30 Aug 2026
  1. Google Search Central, “AI features and your website”, last updated 10 Dec 2025 (indexed and snippet-eligible requirement; first-party): developers.google.com/search/docs/appearance/ai-features
  2. Google Search Central, “Google’s Guide to Optimizing for Generative AI Features on Google Search”, 10 Jul 2026 (no new AI text files needed; first-party): developers.google.com/search/docs/fundamentals/ai-optimization-guide
  3. OpenAI, crawler and bot documentation (OAI-SearchBot, GPTBot, ChatGPT-User; about 24 hours to reflect a robots.txt change; first-party): developers.openai.com/api/docs/bots
  4. Anthropic, “Does Anthropic crawl data from the web?” (ClaudeBot, Claude-SearchBot; first-party): support.claude.com/en/articles/8896518
  5. Perplexity, bots documentation (PerplexityBot, Perplexity-User; user-triggered fetching; firewall allowlisting; first-party): docs.perplexity.ai/guides/bots
  6. Apple, “About Applebot” (Applebot, Applebot-Extended; first-party): support.apple.com/en-us/119829
  7. Zecchini, Moore, Ubl, Siddle, “The rise of the AI crawler”, Vercel, 17 Dec 2024; 404 and redirect shares by crawler (first-party log data): vercel.com/blog/the-rise-of-the-ai-crawler
  8. Allaham & Diakopoulos, 2026; 27.1% of cited URLs could not be scraped, being inaccessible, removed or non-textual (preprint, reported in the survey at note 9)
  9. “Optimizing Visibility in Generative Engines: A Critical Survey of Generative Engine Optimization (2023 to 2026)”, arXiv:2607.14035; Table 5 rates citation scores predicting clicks or revenue at very low confidence: arxiv.org/abs/2607.14035
FAQ

Frequently asked questions.

What is the single most common failure on this list?

Item 03, and specifically an inherited wildcard disallow in robots.txt that predates everybody currently on the team. It removes every search-side crawler at once, it produces exactly the log signature that genuine disinterest produces, and it stays invisible unless the file is evaluated token by token rather than read top to bottom. Check it before spending a day in the logs.

Can I skip the log work if tier one passes?

No, because tier one tests intent while the logs test enforcement. A bot-management rule at the edge blocks a request before your application or your robots.txt is ever consulted, and it is usually owned by a different team. Perplexity's own documentation warns site owners that a web application firewall may need its bots allowlisted explicitly. Tier one cannot see any of that.

How long after a fix should I expect anything to change?

Longer than a day for behaviour, and longer than that for outcomes. OpenAI documents about 24 hours for a robots.txt change to be reflected, and Perplexity documents up to 24 hours, so a same-afternoon retest reads the old state and looks like failure. Crawl volume moves next, over weeks. Whether citations follow is a separate question this list does not answer.

Does passing every item mean I will get cited?

No. The checklist establishes eligibility, meaning the ability to be a candidate at all, and nothing further. Selection, ranking and synthesis all happen after it, and a page can pass all ten items while matching none of the queries the engine actually issued. Read a full pass as an obstacle removed, then go and do the coverage and content work.

Bavior Editorial

The team that researches and maintains Bavior’s writing on Reddit marketing and AI search visibility. Every figure here is attributed to a named source with the date it was checked, and none of our links are affiliate links.

Found a number that looks wrong? Tell us and we will re-check it: support@bavior.com

Eligibility is the floor, not the finish.
See who gets cited once you pass.

Start free trial