Home/Blog/llms.txt checker
Tutorial · AI visibility

llms.txt checker: how do you validate the file, and what does it change?

Seven checks you can run by hand in ten minutes, what Google, OpenAI and Anthropic actually document, and the part of AI visibility no text file reaches.

Updated 20 Sep 202612 min readBavior researchNo affiliate links
Run the free GEO audit Skip to the 7 checks
GEO & AI visibility
On this page
Share this
The short answer

An llms.txt checker confirms that the file at /llms.txt loads, is served as text, and follows the proposed format: one H1, a blockquote summary, and H2 sections of markdown links. It cannot tell you whether any AI engine reads the file, and most never do: a log analysis of 137,210 domains, published 15 Jun 2026, found that 97% of valid files received zero requests in May 2026.3

Key takeaways
  • The file is a proposal from 3 Sep 2024, not a standard. Its only required element is an H1, so “valid” is a low bar.1
  • Google’s own guide says Search ignores the file, and no AI vendor documents its crawlers reading one.25
  • The settings that do gate an AI crawler live elsewhere: robots.txt rules per user agent, server-rendered HTML, and CDN bot rules.68
  • A readable site is a floor. Whether an answer names you depends on the third-party sources the engine cites, which is what Bavior measures and acts on, with your approval on every post.
Definitions

What is llms.txt, and who actually reads it?

The file is a markdown document at the root of a site that hands a language model a short, curated map of the pages worth reading. Jeremy Howard published the proposal on 3 Sep 2024 and last modified it on 10 Aug 2026.1 The format is small: “An H1 with the name of the project or site. This is the only required section”, then a blockquote summary and H2 sections holding link lists.

Read the scope first. The author expected the file “would mainly be useful for inference rather than training”: an agent fetching documentation while it works for a user.1 Getting named in a ChatGPT answer is not a use case the proposal describes.

Adoption and readership point in opposite directions. In the June 2026 log study, 28% of 137,210 domains publish the file, yet 97% of the valid ones got no requests in May 2026, and AI retrieval bots made 1.1% of the requests that did arrive, against 21.7% for SEO audit tools.3 An earlier study of nearly 300,000 domains, published 7 Nov 2025, found 10.13% adoption and no relationship between having the file and being cited.4

The vendors agree. Google’s guide to generative AI features, last updated 10 Jul 2026, names the file directly. Creating one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them”.2 John Mueller was blunter on r/TechSEO, as reported on 6 Jul 2026: “I don’t know of any other crawler / llm confirming they’re using these”.5 OpenAI and Anthropic document robots.txt as the control, and neither documents a crawler consuming the file.67

Our evidence review, Length, schema and llms.txt, covers every study so far. What the record proves: the format is real and easy to satisfy. What it leaves unsolved: a reader.

The tool

What does an llms.txt checker actually test?

A checker, sometimes sold as a validator, is a linter. It fetches one URL and compares the response with the proposal’s grammar: the status code, an H1 on line one, a blockquote, the link syntax, and sometimes whether each link resolves. Like an HTML validator, it catches accidents.

The accidents are common. A single-page app answers every unknown path with its HTML shell and a 200, so /llms.txt “exists” and contains a login screen. A generator run against staging writes staging URLs into production. The June 2026 log study had to screen out soft 404s and HTML masquerading as markdown before it could count valid files.3

Two questions sit outside any checker. Did anything request the file? Only your server logs know. Are you named in the answers your buyers read? A green result is a claim about syntax, never about visibility.

Priya, a solo founder selling a $49 per month scheduling tool for salons, is an illustrative composite of the pattern. She shipped a 212-link generated file and a validator turned green. Six weeks later her logs showed 41 requests to /llms.txt: 38 from SEO audit tools, 3 from unidentified bots, none from an AI retrieval bot. The ChatGPT answer for “best scheduling software for salons” still named three competitors and cited two Reddit threads. The file was valid. Nothing she cared about had moved.

What Bavior does not have

Bavior does not ship an llms.txt checker, and the free GEO audit does not validate the file. We would rather hand you the seven checks below than score a file the evidence says goes unread. We publish one on bavior.com and expect nothing from it.

Tutorial

How do you check an llms.txt file by hand?

Seven checks, a terminal and a text editor, about ten minutes. The first two catch most broken files.

CheckHow to run itIt passes when
1. The file exists at the rootRequest the headers: curl -sI https://yourdomain.com/llms.txtHTTP 200, with no redirect to the homepage or a login page
2. It is served as textRead the content-type header in that responsetext/plain or text/markdown, never text/html. The proposal names no MIME type; log studies screen out fake files this way3
3. One H1 opens the fileOpen the file and read line oneExactly one line starting with a single #, naming the site: the only required element1
4. A blockquote summary followsLook for a line starting with >One or two sentences a stranger could repeat to say what you sell
5. Link lists sit under H2sRead each entry against - [name](url): noteNo bare URLs, no HTML tags, no headings nested inside a list
6. Every link resolvesExtract the URLs and request each one (command below)All 200, all production URLs, none disallowed in your robots.txt
7. The file is curated and currentCompare it with your sitemap and your last releaseSkippable links sit under an H2 named Optional; no retired products or old prices
 
# Acme Scheduling

> Appointment scheduling for salons and clinics. Plans from $49 per month.

## Docs
- [Quick start](https://acme.example/docs/quick-start.md): setup in ten minutes
- [API reference](https://acme.example/docs/api.md)

## Optional
- [Changelog](https://acme.example/changelog.md)
grep -o 'https://[^)]*' llms.txt | xargs -n1 curl -s -o /dev/null -w "%{http_code} %{url_effective}\n"

The first block is a minimal passing file. The second is the one-liner for check six: it prints a status code beside every URL. What the seven checks prove: the file is well formed, reachable and honest about your site. What they leave unsolved: whether any crawler will ask for it.

Crawler access

Which AI crawler settings matter more than the file?

01

Is each AI crawler allowed in robots.txt?

OpenAI documents three agents: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for actions a user starts. Its wording is direct: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers”, and a change can take about 24 hours to register.6 Anthropic lists ClaudeBot, Claude-User and Claude-SearchBot and says they honor robots.txt.7

02

Is the content in the HTML before JavaScript runs?

A network-level analysis published 17 Dec 2024 found that “none of the major AI crawlers currently render JavaScript”. ChatGPT’s crawler fetched JavaScript files in 11.50% of requests and Claude’s in 23.84%, and neither ran them.8 View source and search for a sentence you can see on screen.

03

Is your CDN blocking what robots.txt allows?

On 1 Jul 2025 Cloudflare announced it was “changing the default to block AI crawlers unless they pay creators for their content”.9 Your robots.txt can say yes while the edge says no, so read your CDN’s bot settings beside the file.

Owen, the head of marketing at a 30-person logistics SaaS, is a second illustrative composite. His file passed every check above. His robots.txt also carried a Disallow for OAI-SearchBot, pasted in by a contractor in 2024, and nobody remembered it. For 19 months the company had opted out of ChatGPT search while paying an agency $2,500 a month for “AI SEO”. The fix took four minutes.

What these checks prove: an AI crawler can reach and read your pages. What they leave unsolved: whether the engine has any reason to pick you.

The part a file cannot reach

What decides whether AI engines name you?

The sources the engine retrieves, and for buyer questions those are mostly not your pages. A study published 10 Nov 2025, covering 248,000 Reddit URLs across 217,000 prompts, found ChatGPT search included Reddit links in 13% of its responses, Google AI Mode in 9% and Perplexity in 4%, and most cited posts had fewer than 20 upvotes.10 A small thread that answers the question gets quoted. A valid file makes you readable; it is not in that race.

This is the problem Bavior is built for. It runs a fixed set of your buyer prompts across ChatGPT, Gemini, Perplexity and Google AI Overviews on a schedule and records, for every answer, whether you were named and which pages were cited. Two numbers come out: mention share, the portion of answers that name you, and citation share, the portion of cited sources that are yours.

The Top Cited Domains card lists the sites the engines cite most, each with a Type label such as UGC, Reference, Editorial or Competitor.

Bavior overview cards: Top Cited Domains with a Type column labelling reddit.com as UGC and other sites as Reference, Competitor, Editorial or Other, Top Mentioned Prompts with a mention percentage for each prompt, and an AI Models ring showing 180 scans split across ChatGPT, Gemini, Google AI Overview and Perplexity
Sample data from a demo workspace; every figure is a demo figure. Red box: the Type column.

Open one prompt and Citation Events lists each answer with its engine, country, date, a Mentioned or Not mentioned label, the cited page, and a View answer button. That is the evidence a checker cannot produce: which page the engine leaned on, and whether your name was in the answer.

Bavior prompt detail: citation counts by engine for ChatGPT, Gemini, Perplexity and Google AI Overview, a by-country card, and two Citation Events, one from Google AI Overview labelled Mentioned and one from Perplexity labelled Not mentioned, each with its date, cited page title, answer excerpt and a View answer button
Sample data from a demo workspace, with fictional product names. Red boxes: the Mentioned and Not mentioned labels.

Then comes the half a dashboard skips. Where a cited source is a live Reddit thread, Bavior reads the subreddit’s rules, drafts a reply in your voice, and places it in an approval queue. Nothing publishes until you approve it, and an approved reply goes out from one of Bavior’s aged high-karma accounts or from your own account. Where a subreddit requires disclosure of a commercial affiliation, disclose it. The AI visibility page covers the measurement side.

Lena, a growth lead at a 12-person help desk startup, is an illustrative composite of how the halves meet. Her 20 buyer prompts named her product in 3 answers, a 15% mention share, and Reddit threads led the cited domains. Over five weeks she approved 9 of 14 drafted replies. On the next run, 7 of 20 answers named her. No tool, this one included, can promise a place in a given answer. What moves is the share across a prompt set.

What this proves: which sources your category’s answers are built from. What it leaves unsolved: the engines’ selection rules, which none of them publish.

Making the file

Should you use an llms.txt generator?

Yes, if you want the file at all, and then edit what it gives you. A generator crawls your sitemap and writes an entry for every URL. That is the opposite of a curated map: a 400-link file tells an agent nothing about which ten pages matter.

Four edits make the output worth shipping. Cut the list to pages a stranger needs: pricing, docs, comparisons, policies. Write the blockquote yourself. Move changelogs under Optional. Then run the seven checks again, since generators are where staging URLs come from.

Budget one hour, after the crawler access fixes, not before. What a generator proves: the file is cheap to produce. What it leaves unsolved: everything here that is not syntax.

GEO audit

What does a GEO audit tool check, and where does it fit?

A GEO audit asks whether an engine can fetch, parse and quote your pages, so it sits between the file check and the answer measurement. The free Bavior GEO audit runs inside a free account: you submit a domain, it fetches your key pages, and it scores each out of 100 on structured markup, content structure, meta and citation density. Every issue comes back with its fix, and the report arrives by email. It is not instant.

It does not score llms.txt, robots.txt, JavaScript rendering or CDN rules, which is why those checks are in this article. One run is free. Weekly re-runs are part of a paid plan, from $99 per month billed monthly or $79.17 per month billed yearly, as of 20 Sep 2026.

Order of work

What should you fix this week, and in what order?

  1. Crawler access. Search robots.txt for the six user agents above, then read your CDN bot rules.
  2. The view-source test. If your pricing sentence is missing from the raw HTML, fix that before anything else.
  3. A GEO audit on your key pages. Fix the high-severity issues and re-score.
  4. A mention share baseline. Record who gets named across 15 to 20 buyer prompts. The free AI visibility checker gives a first reading.
  5. The cited threads. Answer what is asked, with every reply approved before it goes out.
  6. The file, last. Generate it, edit it, run the seven checks, move on.

The proposal reached a second version on 10 Aug 2026, so this order reads today’s evidence. If a major engine documents reading the file, step six moves up. Until then a checker is a ten-minute job, and the hours belong to the sources your buyers’ answers are built from.

Sources: all fetched and checked 20 Sep 2026. Named operators are illustrative composites, not customers.
  1. Jeremy Howard, “The /llms.txt file”, published 3 Sep 2024, modified 10 Aug 2026: llmstxt.org
  2. Google Search Central, guide to optimizing for generative AI features, last updated 10 Jul 2026: developers.google.com/search/docs/fundamentals/ai-optimization-guide
  3. Ahrefs, “We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read”, published 15 Jun 2026: ahrefs.com/blog/llmstxt-study
  4. SE Ranking, LLMs.txt and AI citations study, nearly 300,000 domains, published 7 Nov 2025: seranking.com/blog/llms-txt
  5. Search Engine Journal, Roger Montti, published 6 Jul 2026, quoting John Mueller on r/TechSEO: searchenginejournal.com
  6. OpenAI, crawler documentation (GPTBot, OAI-SearchBot, ChatGPT-User): developers.openai.com/api/docs/bots
  7. Anthropic, crawler support article (ClaudeBot, Claude-User, Claude-SearchBot): support.claude.com
  8. Vercel, “The rise of the AI crawler”, published 17 Dec 2024: vercel.com/blog/the-rise-of-the-ai-crawler
  9. Cloudflare, Matthew Prince, “Content Independence Day”, published 1 Jul 2025: blog.cloudflare.com
  10. Semrush, “We Analyzed 248K Reddit Posts”, published 10 Nov 2025: semrush.com/blog/reddit-ai-search-visibility-study

Continue reading

Learn GEO Length, schema and llms.txt: what tests show

Length, schema and llms.txt for founders: what has actually been tested for AI citation, by whom, and what each test structurally could not see.

 Product Free GEO audit

Review page structure, metadata and citations for your AI search research. What can a technical GEO review establish?

 Product Free AI visibility checker

Explore brand mentions and the sources cited in AI-generated answers. What should an AI visibility check record?

 Blog Best GEO tools compared

The best GEO tools for AI search visibility, compared for founders: 11 tools, prices re-read 30 Aug 2026, and what the evidence actually supports.

 Blog AI visibility: monitoring vs execution

An AI visibility monitoring tool scores you, it does not move you. For founders: what the score leaves undone, and 4 execution routes that change it.
FAQ

Frequently asked questions.

What is an llms.txt checker?

An llms.txt checker is a linter for one file. It requests /llms.txt, confirms a 200 response served as text, and compares the content with the proposed format: one H1, a blockquote summary, and H2 sections of markdown links. It says nothing about whether an AI crawler ever requests the file.

Does Google use llms.txt?

No. Google's guide to generative AI features, last updated 10 Jul 2026, says Google Search ignores these files, so creating one will neither harm nor benefit your visibility or rankings. John Mueller repeated the point on r/TechSEO in July 2026 and added that he knew of no other crawler confirming use.

Is llms.txt the same as robots.txt?

No. A robots.txt file sets access rules that crawlers from OpenAI, Anthropic and Google document and honor. The other file is a proposed markdown map of your best pages, and no major engine documents reading it. A mistake in robots.txt can remove you from ChatGPT search. A mistake in the map changes nothing measurable.

Do I need an llms.txt generator?

Only if you have decided to publish the file. A generator saves typing, but it lists every URL in your sitemap, which defeats a curated map. Cut the output to the pages a stranger needs, write the summary yourself, then run the seven checks. Budget one hour, after your crawler access fixes.

Does the Bavior GEO audit check llms.txt?

No. The free Bavior GEO audit scores each page out of 100 on structured markup, content structure, meta and citation density, and emails the report. It does not score llms.txt, robots.txt, JavaScript rendering or CDN rules. Those take a minute each to check by hand, and the steps are in this article.

Bavior Editorial