By KEY TAKEAWAYS
- llms.txt is a proposed Markdown file at your site's root that gives AI systems a curated map of your best pages; it is a proposal, not an official standard.
- Google's John Mueller has said none of the AI services he knows of use it, and Google's own AI search guidance says you don't need special AI text files, so don't expect it to move rankings or citations.
- It costs about an hour to add and can help developer-style use cases like documentation and coding assistants, so treat it as cheap housekeeping after your real GEO basics are done.
Every few months, the SEO world discovers a tiny text file and decides it will change everything. Remember the keywords meta tag? Right.
llms.txt is the 2020s version of that story, with one twist: it isn’t useless, it’s just wildly oversold. If you’ve been told you must add one “or AI will ignore you”, this post is the calm second opinion. We’ll cover what it is, who actually reads it, how to build one in an hour, and where it sits in a real AI-search plan.
What is llms.txt?
llms.txt is a Markdown file at your site’s root that gives large language models a curated map of your most useful pages. Jeremy Howard of Answer.AI proposed it in September 2024 as a way to give models a compact entry point instead of making them parse cluttered HTML (Answer.AI).
The format is deliberately simple. Per the spec, the file starts with an H1 (your site or project name), usually a short blockquote summary, then H2 sections containing lists of links with one-line descriptions. A section called “Optional” marks links a model can skip if space is tight.
Think of it as a table of contents written for machines, not a rulebook. That distinction matters in the next section.
Who actually reads llms.txt?
Today, there’s no public confirmation that the major AI search products use it. It’s a proposal, not an official web standard, and adoption by model providers is unproven.
What we can say, with sources:
- John Mueller of Google said that, as far as he knows, none of the AI services have said they use llms.txt, and he compared it to the keywords meta tag: a claim about what a site is about that search engines can simply check for themselves (Search Engine Journal).
- Google’s official guidance on AI Overviews and AI Mode says site owners don’t need special files or AI-only rewrites to show up there. The advice is to stick to fundamentals like crawlable, useful, original pages (Google Search Central).
Where it plausibly helps: developer documentation and coding assistants. A tool that fetches docs on demand benefits from a clean index of Markdown pages, which is the use case the proposal grew out of.
Our read: for a marketing site, llms.txt is a low-cost, low-evidence bet. Do it for tidiness, not because you expect a ranking jump.
llms.txt vs robots.txt vs sitemap.xml: what’s the difference?
They do different jobs. robots.txt controls crawler access, sitemap.xml lists URLs for search engines, and llms.txt (optionally) suggests your best pages in a readable summary. Only the first two have broad, confirmed support.
| File | Purpose | Format | Confirmed support |
|---|---|---|---|
| robots.txt | Allow or block crawlers | Plain text rules | Universal among well-behaved bots |
| sitemap.xml | List URLs for indexing | XML | Google, Bing and others |
| llms.txt | Curated guide to key pages | Markdown | Unconfirmed from major AI providers |
| llms-full.txt (optional) | Full text of key content in one file | Markdown | Unconfirmed |
One practical consequence: if you want to block AI training or retrieval crawlers, llms.txt can’t do it. Use robots.txt and your CDN or firewall rules for that.
Should your website have one?
Add one if it takes under an hour and you’ve already handled the basics; skip it if it would distract from them. Here’s the decision framework we use with clients:
| Your situation | Our recommendation |
|---|---|
| Developer docs, API, SaaS with a help center | Yes. Docs sites are the best-fit use case. |
| Content-heavy marketing site with 50+ pages | Probably. It forces you to pick your best pages. |
| Small brochure site with 5-10 pages | Optional. A clear nav and sitemap already do the job. |
| Clean fundamentals missing (no schema, thin pages, slow site) | Not yet. Fix those first. |
| You want to block AI crawlers | Use robots.txt and server rules instead. |
The hidden benefit is the exercise itself. Choosing the 15 pages that best describe your business is a surprisingly good strategy workshop. In our experience, teams struggle to list them, and that’s a sign the site structure needs work.
How to add llms.txt in six steps
You need a text editor and access to your site root. Most people finish in 30 to 60 minutes.
- List your best pages. Services, pricing, key guides, case studies, about and contact. Aim for 10 to 30 links, not your whole site.
- Write a one-line description for each. Plain English, what the page answers.
- Group them under H2 headings such as Services, Guides, Company.
- Add a short summary (two or three sentences) under the H1 explaining who you are and who you serve.
- Publish at
/llms.txtand confirm it loads as plain text (not a styled HTML page). - Automate updates so new posts appear without manual edits. Many CMSs and static site generators can generate it at build time. (This site’s own file is generated that way, so it never goes stale.)
Here’s a minimal template:
# Your Company Name
> One or two sentences on what you do and who you do it for.
## Services
- [SEO and AI search](https://example.com/services/seo/): Technical SEO, content and AI-search optimization for B2B brands.
- [Paid ads](https://example.com/services/ads/): Meta and Google campaigns for e-commerce.
## Guides
- [How AI search works](https://example.com/blog/ai-search/): A plain-English explainer on how assistants pick sources.
## Company
- [About](https://example.com/about/): Team, approach and contact details.
## Optional
- [Full blog archive](https://example.com/blog/): Everything we've published.What are the common llms.txt mistakes?
Most problems come from treating it like a sitemap or a sales page. Here’s what we’d avoid:
- Dumping every URL. The point is curation. If it lists 800 pages, you’ve rebuilt a sitemap, badly.
- Marketing copy in the descriptions. “World-class solutions” tells a model nothing. “Pricing for our three SEO plans” does.
- Blocking the file or serving it as HTML. It should load as plain text with a normal 200 status.
- Letting it go stale. Links to deleted pages are worse than no file. Generate it automatically or review it quarterly.
- Hiding content you don’t want used. The file doesn’t protect anything. If a page is private, put it behind a login, not a Markdown list.
- Expecting a ranking boost. Nothing in Google’s published guidance suggests one.
What about llms-full.txt?
llms-full.txt is an optional companion that puts the full text of your key pages in one Markdown file. It follows the same idea: fewer tokens wasted on layout, more on content. The upside is for tools that load documentation into a model’s context in one go. The downside is size: a big site can produce a huge file, and nobody has confirmed that major assistants fetch it. Our advice: only build it if you run docs or a knowledge base, and keep it automated so it can’t drift from the live pages.
How do you know if llms.txt is doing anything?
Check your server logs. Search for requests to /llms.txt and note the user agents. If you see AI crawlers fetching it, you have evidence. If you only see SEO audit tools and your own browser, you have your answer.
Better still, measure what matters: whether your brand appears in AI answers for the questions your buyers ask. Run a fixed set of 10 to 20 prompts in ChatGPT, Perplexity, Gemini and Google AI Mode each month and record whether you’re cited. That tells you far more than a file ever will. Our GEO guide walks through the tracking method.
What actually earns AI citations
Clear, crawlable, trustworthy pages. That is the unglamorous truth, and it matches Google’s own advice. If you want assistants to cite you, spend the hour on these first:
- Answer-first pages that state the answer in the opening lines of each section.
- Original information (data, examples, first-hand experience) that other pages can’t copy.
- Accurate, consistent facts about your business across your site and profiles.
- Pages that aren’t blocked by robots.txt, login walls or heavy scripts.
For the bigger picture, see SEO vs GEO vs AEO, and if you’d like us to audit this for you, that’s exactly what our SEO and AI search service covers.
What to do this week
- Audit the basics first. Run through our free AI Search (GEO) Checklist and fix anything marked red.
- Pick your 15 best pages and write one-line descriptions. If this is hard, note it: your site structure may be the real problem.
- Publish llms.txt using the template above (or generate it automatically at build time) and confirm it loads as plain text.
- Add a log check to your calendar for 30 days from now: who requested
/llms.txt? - Start a citation tracker: 10 buyer questions, four AI tools, one spreadsheet, checked monthly.
A tiny file won’t make you the answer. A clear, useful website might.
Frequently asked questions
What is llms.txt?
+
llms.txt is a plain Markdown file placed at the root of a website (yourdomain.com/llms.txt). It was proposed in September 2024 by Jeremy Howard of Answer.AI. It gives large language models a short, curated summary of the site and links to the most useful pages, so a model doesn't have to wade through navigation, scripts and ads.
Does Google use llms.txt?
+
No evidence says it does. John Mueller said that, as far as he knows, none of the AI services have said they use llms.txt, and compared it to the keywords meta tag. Google's guidance on generative AI features also says site owners don't need special AI text files to appear in AI Overviews or AI Mode.
Will llms.txt help my brand get cited by ChatGPT or Perplexity?
+
There's no public confirmation that ChatGPT, Perplexity, Claude or Gemini read llms.txt when answering questions. Citations still come from crawlable, clear, trustworthy pages. If you add the file, treat any benefit as a bonus and measure it in your server logs rather than assuming it works.
How do I add llms.txt to my website?
+
Create a Markdown file named llms.txt with an H1 site name, a one-line blockquote summary, and H2 sections of links with short descriptions. Upload it to your site root so it loads at yourdomain.com/llms.txt, check it returns plain text, and update it whenever you publish key pages. Most CMS plugins can automate this.
What is the difference between llms.txt and robots.txt?
+
robots.txt tells crawlers what they may or may not fetch, and well-behaved bots follow it. llms.txt doesn't control access at all. It is an optional, human-readable guide that suggests which pages best describe your site. One is a gatekeeper with real support, the other is a suggestion box that few systems check.
Sources
- Answer.AI (Jeremy Howard), /llms.txt: a proposal to provide information to help LLMs use websites (September 2024)
- llmstxt.org, The /llms.txt file
- Search Engine Journal, Google's Mueller: llms.txt is comparable to the keywords meta tag
- Google Search Central, AI features and your website
WRITTEN BY
Junaid Mubashar
Founder of an AI-native growth agency. Junaid plans and runs full-funnel marketing (SEO and AI search, paid social, Google Ads) and builds the AI workflows and agents that automate it. Everything on this blog is tested with real campaigns before we recommend it.
🔍 The AI Search (GEO) Checklist
42 checks to get your brand cited by ChatGPT, Perplexity, Gemini and Google AI Overviews.