In Brief
- Core Answer: llms.txt is a proposed Markdown file at your site root that offers language models a curated map of your most useful content. It is a community proposal, not a standard, and no major AI provider has confirmed using it as a ranking or retrieval input.
- Why It Matters: It costs an hour to implement and is being sold as a GEO strategy. Both facts are worth knowing before you prioritise it.
- Best For: Technical marketers and SEO leads deciding whether llms.txt deserves a place on the roadmap.
llms.txt is a Markdown file placed at the root of a domain that lists a site's most important pages with short descriptions, intended to give language models a clean, curated entry point instead of forcing them to crawl and parse navigation-heavy HTML. It was proposed in 2024 by Jeremy Howard of Answer.AI. Adoption by publishers has been enthusiastic; adoption by the AI providers it is aimed at has not been confirmed.
That gap is the whole story, and it is worth understanding before you either dismiss the file or build a strategy around it.
What an llms.txt File Actually Contains
The proposal specifies Markdown, not a machine format like XML or JSON, on the reasoning that language models parse Markdown natively. A conforming file has:
- An H1 with the site or organisation name.
- A blockquote giving a short summary of what the site is.
- Optional paragraphs of additional context.
- H2-delimited sections containing lists of links, each with a brief description.
- An optional "Optional" section marking content that can be skipped when context is limited.
A companion convention, llms-full.txt, holds the expanded content itself rather than links to it — effectively a single-file version of the site's substantive text.
Neither file carries directives. There is nothing in the specification equivalent to robots.txt's Allow and Disallow. It is a suggestion, not an instruction.
What It Is Not
Three confusions are worth clearing up, because they drive most of the bad advice.
- It is not robots.txt for AI. robots.txt controls access and is honoured by well-behaved crawlers. llms.txt controls nothing. If you want to permit or block GPTBot, PerplexityBot or ClaudeBot, that is a robots.txt and user-agent question, entirely separate from this file.
- It is not a sitemap. A sitemap is consumed by search engines as a crawl aid and its use is documented. llms.txt is aimed at inference-time consumption by models, and there is no published evidence of any major provider fetching it during retrieval.
- It is not a ranking signal. No provider has stated that its presence affects selection. Claims that adding it improves AI visibility are, at present, unsupported.
The Adoption Question
Here is the honest position as it stands. A substantial number of documentation sites, developer tools and SaaS companies now publish llms.txt — it has become close to a convention in developer documentation, where the format's origins lie. Some AI coding tools and documentation assistants do read it, which is a genuine use case.
What has not happened is confirmation from the large consumer-facing providers that their search or retrieval pipelines fetch it. Google has publicly indicated that it does not use llms.txt for search, and the file is absent from published crawler documentation for the major assistants. Server log analyses shared by practitioners have generally shown very low request volume for the file relative to normal crawling.
So: a real convention with a real niche, being marketed as something broader.
Should You Publish One?
Yes, with correctly calibrated expectations. The cost-benefit is straightforwardly favourable:
Cost | Benefit |
|---|---|
An hour to write, minutes to update | Consumed today by documentation assistants and coding tools |
One more file to keep current | Free option value if adoption broadens |
No SEO risk if correctly formatted | Forces a useful editorial exercise |
That last row is underrated. Writing the file requires deciding which twenty pages actually represent your site and describing each in one sentence. Most teams cannot do this without argument, and the argument is worth having regardless of whether any model reads the output.
What it should not do is displace work on the layers that demonstrably matter — retrievability, question ownership, entity clarity and third-party consensus. If llms.txt appears above those in a proposal, the proposal is optimising for what is easy to deliver.
How to Write One That Is Actually Useful
- Curate ruthlessly. Fifteen to forty links, not your whole sitemap. The point of the file is editorial selection; a dump of every URL defeats it.
- Write real descriptions. One sentence per link saying what the page answers, not what it is called. "How to measure revenue influenced by AI search, including GA4 configuration" beats "Attribution Guide".
- Group by user intent. Sections like "Getting started", "Reference", "Guides", "Company" are more useful than your navigation labels.
- Include the company facts. A short block covering what you are, what category you belong to, founding details and how to contact you. If anything in the file gets read and reused, this is the part that reduces the chance of being described wrongly.
- Keep it current. A stale llms.txt pointing at retired pages is worse than none, because it is the version of your site a machine may treat as authoritative.
- Date it. A "last updated" line costs nothing and signals maintenance.
If you publish llms-full.txt as well, keep it genuinely full — the expanded text, not a longer list of links — and accept that it will be large.
How to Tell If Anything Reads It
Check your server logs. Filter requests to /llms.txt and /llms-full.txt and group by user agent. You will see the answer for your own site within a month, which is more reliable than any general claim including this one. If you see meaningful volume from an agent you care about, that is a reason to invest more in the file. If you see almost nothing, you have spent an hour and learned something.
This is the correct posture toward the whole category of proposed AI conventions: implement the cheap ones, instrument them, and let your own logs settle the argument.
What to Do Instead, If You Only Have One Afternoon
If the goal is AI visibility and the budget is one afternoon, llms.txt is not where it goes. In order of return:
- Audit robots.txt against the current list of AI user agents. A blocked agent is a total exclusion from that surface and it is common.
- Confirm your key pages render server-side and are indexed.
- Rewrite the opening paragraph of your five highest-intent pages so each answers its heading in fifty self-contained words.
- Publish one canonical facts page stating plainly what your company is, what it does, and what it costs.
- Then, if time remains, write llms.txt — using the page list you just clarified.
Done in that order, the file is a byproduct of useful work rather than a substitute for it.
The Broader Pattern Worth Recognising
llms.txt is the first of a class of proposals that will keep appearing: lightweight, publisher-side conventions that promise influence over AI systems in exchange for a small implementation cost. Others in circulation include content-licensing signals, AI-specific meta directives, and various schemes for declaring training permissions.
They share a structural weakness. A convention only has force if the consuming party agrees to honour it, and the consuming parties here have limited incentive to bind themselves. robots.txt worked because search engines needed publisher cooperation to build an index that publishers wanted to be in. The bargain in AI retrieval is less symmetrical, and the conventions reflect that.
The practical rule that follows: adopt cheap conventions, do not restructure work around them, and instrument everything so your own logs tell you which ones acquired force. That posture costs almost nothing and it is right regardless of how the standards question resolves.
Common Implementation Mistakes
- Serving it with the wrong content type. It should be plain text or Markdown, served at
/llms.txtwith a 200 status. A file behind a redirect chain, or returned as HTML by a catch-all route, may not be usable. Check with a direct request, not a browser. - Generating it from the sitemap. An automatically generated list of every URL removes the only thing that makes the file useful — curation — and produces something larger and less informative than the sitemap it came from.
- Descriptions copied from meta descriptions. Meta descriptions are written to earn clicks. The description here should say what the page answers, in plain declarative language, with no promotional framing.
- Forgetting it exists. This is the common one. A file written once and never updated becomes a list of dead links, and it is the only version of your site some tools will see. Put it in the same release checklist as the sitemap.
- Treating it as a place for claims. The file is a map, not a pitch. Marketing language in it does not improve retrieval and does make the file less useful for the tools that genuinely consume it.
Where It Fits in a Real Programme
If you are building an AI visibility programme, llms.txt belongs in the technical-hygiene tier alongside structured data validity and canonical consistency: worth having correct, never worth a status update. The tiers that produce measurable movement are retrievability, question ownership, entity clarity and third-party consensus — set out in the GEO strategy framework — and none of them are affected by this file.
The reasonable summary is therefore unexciting and, for once, uncontested by the evidence: publish it, keep it accurate, check your logs, and spend the rest of the afternoon on something that moves.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed Markdown file at a website's root that gives language models a curated list of the site's most important pages with short descriptions, intended as a clean alternative to parsing navigation-heavy HTML.
Does llms.txt actually work?
It is read by some documentation assistants and coding tools. No major consumer AI provider has confirmed using it in search or retrieval, and Google has indicated it does not use it for search. Treat it as low-cost housekeeping rather than a visibility lever.
Is llms.txt the same as robots.txt?
No. robots.txt controls crawler access and is honoured by compliant crawlers. llms.txt contains no directives and controls nothing — it only offers a curated content map.
What is the difference between llms.txt and llms-full.txt?
llms.txt lists links with short descriptions. llms-full.txt contains the expanded content itself in a single file, so a model can read the substance without following links.
Does llms.txt help SEO?
There is no evidence it affects search rankings, and Google has said it does not use it for search. It carries no SEO risk when correctly formatted, but it should not displace indexation, content structure or entity work.
How do I create an llms.txt file?
Write Markdown with an H1 site name, a blockquote summary, then H2 sections listing fifteen to forty key pages, each with a one-sentence description of what it answers. Save it as llms.txt at your domain root and keep it updated.