llms.txt: What It Is and How to Add One (2026 Guide)

Spend any time in an SEO forum this year and you will have seen the argument. One person swears that adding an llms.txt file to your website boosts your visibility in ChatGPT, Perplexity and Google’s AI Overviews. The next swears it does nothing at all. Both are partly right, and that is exactly the problem.

This guide explains what llms.txt actually is, where it came from, how to format one properly, and just as importantly, what it can’t do for you. It draws on what the file’s own creator and Google have each said in public. AI assistants and AI-powered search now shape how many people find web content, so it pays to know exactly what this file influences and what it doesn’t.

What is llms.txt?

llms.txt is a proposed standard. It’s a plain Markdown file placed at the root of a website (yourdomain.com/llms.txt) that gives large language models (LLMs) a short, curated summary of a site’s most important content. Think of it as a cleaned-up index of your pages, written for a model rather than a human scrolling your homepage.

The idea was proposed on 3 September 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai, at llmstxt.org. His reasoning was practical rather than promotional: language models have limited context windows, and a typical web page, full of navigation menus, ads and JavaScript, is a wasteful and imprecise way for a model to learn what a site actually contains. llms.txt was designed to hand models a clean shortcut instead.

llms.txt vs robots.txt vs sitemap.xml: what’s the difference?

These three files sound similar but do very different jobs.

• robots.txt tells a web crawler what it’s allowed to access. It’s a permissions file that governs crawling and indexing.
• sitemap.xml lists every indexable URL on your site, so search engines understand the full scope of your pages. It’s a completeness file.
• llms.txt is a curated summary of your most important pages, written to be efficient for a language model to read. It’s a priority file, not a permissions or completeness file.

None of the three replaces the others. A well-maintained site usually has all three, each doing its own job.

The format: what actually goes in an llms.txt file

The specification is deliberately minimal. Unlike heavier structured data formats such as schema.org markup, it’s built to be read and edited by a human, not just parsed by machines. According to the official llmstxt.org specification, a valid file follows this structure:

1. An H1 heading with your site or project name. This is the only strictly required element.
2. A blockquote summary, a single sentence describing what the site or project does.
3. Optional free-text context, a short paragraph with any extra detail worth knowing.
4. H2 sections grouping links to your most important pages, each formatted as a Markdown link with a short description.
5. An “Optional” H2 section at the end, for secondary content that’s useful but not essential.

Here’s a simplified example:

# Aus Asia Online

> Aus Asia Online is an Australian digital marketing agency providing SEO, content marketing and AI search optimisation.

## Services
- [SEO Services](https://www.ausasiaonline.com.au/search-engine-optimisation/): Full-service SEO campaigns for Australian businesses.
- [AI SEO](https://www.ausasiaonline.com.au/ai-seo/): Optimising content for AI-driven search and AI Overviews.
- [Generative Engine Optimisation](https://www.ausasiaonline.com.au/generative-engine-optimisation/): Getting your content cited by AI systems.

## Optional
- [Contact](https://www.ausasiaonline.com.au/contact/): Book a strategy session with the team.

The specification also suggests linking to .md (Markdown) versions of pages where possible, since plain Markdown is cheaper and more precise for a model to parse than rendered HTML. No Markdown versions of your pages? Linking to your regular URLs is fine. The standard is flexible on this point.

How to create an llms.txt file for your website

1. List your most important pages. Pick the handful that best represent what your site or business does. Don’t dump your entire sitemap in here. The whole point is curation, not completeness.
2. Write a one-sentence summary. This blockquote is what a model uses to grasp your site’s purpose, so make it specific rather than generic marketing language.
3. Group your links under clear H2 headings. Services, Documentation, Guides, About, whatever categories make sense for your content.
4. Save the file as llms.txt and upload it to your root directory, so it’s reachable at yourdomain.com/llms.txt.
5. Keep it current. A stale file that still points to pages you deleted six months ago is worse than no file at all.
6. Check your robots.txt isn’t blocking AI crawlers. A robots.txt that disallows bots like GPTBot or ClaudeBot will also stop them reading your llms.txt, no matter how well-formatted it is.

Prefer not to build the file by hand? Several free llms.txt generator tools can create one from your sitemap in a couple of minutes. Whichever way you produce it, treat step 5, keeping it current, as a recurring task rather than a one-off.

Does llms.txt actually improve AI search visibility?

This is the part most articles gloss over, and it’s the most important part.

Jeremy Howard’s original proposal was never about search rankings or AI Overview citations. It was written to help LLM-powered coding tools and AI agents work more efficiently with technical documentation, the kind of case where a tool like Cursor or Claude Code needs quick access to a project’s API reference. Applying that idea to a general business or content website repurposes the standard rather than using it as intended.

Google has addressed this directly. Its guide to optimising for generative AI features states plainly that website owners don’t need to create new machine-readable files, AI text files or Markdown formats to appear in generative AI search. As of mid-2026, no major AI provider, including OpenAI, Google, Anthropic or Perplexity, has publicly confirmed that its systems treat llms.txt as a ranking or citation signal.

That doesn’t make the file useless. It makes it a documentation aid, not an SEO tactic. Run a software product, an API or developer documentation aimed at people using AI coding tools, and an llms.txt file can help those tools work with your content more efficiently. Run a local business, an online store or a content site trying to improve its AI visibility, and your time is far better spent on the things that clearly help: clear content structure, structured data that helps search engines understand what a page is about, and expert-led pages that also perform well in traditional search results.

Good, well-organised content and a clear user experience are what help both people and machines understand your site. An llms.txt file is a nice-to-have on top of that foundation, not a substitute for it.

For a deeper look at what does move AI visibility, our guide on ranking in Google AI Overviews covers the content structure, trust signals and technical foundations Google has actually confirmed matter. Weighing up llms.txt against a broader AI content strategy? Our explainer on Generative Engine Optimisation (GEO) walks through what actually helps content get cited by AI systems, and what’s mostly noise.

Should you bother adding one?

A short, honest answer: it depends on your site.

• Yes, worth doing if you run developer documentation, an API, an open-source project or any site consumed mainly by AI coding tools and agents. This is exactly the case llms.txt was built for.
• Low priority, but harmless if you run a general business or content website. It won’t hurt anything and it’s cheap to set up, but don’t expect it to shift your AI Overview visibility. Your time works harder on solid technical SEO and content built to be understood by both people and AI systems, the approach covered in our comparison of AI search vs traditional search.
• Track your actual visibility instead of guessing. Rather than assuming a file is helping, our comparison of AI visibility tools shows how to check whether AI systems cite your business at all.

Frequently asked questions

Is llms.txt an official web standard?

No. It’s a community-proposed convention, not a ratified standard from a body like the IETF or W3C. Adoption is voluntary, and it’s implemented by individual sites and tools that choose to support it.

Does Google use llms.txt for AI Overviews?

Google has stated that no special AI text files are required to appear in its generative AI search features, and it hasn’t confirmed llms.txt as a ranking or citation input.

What’s the difference between llms.txt and llms-full.txt?

llms.txt is a short, curated index of key pages. llms-full.txt, used by some tools though not formally part of the original specification, is a longer file that concatenates fuller page content into a single document.

Will adding llms.txt hurt my SEO?

No. It won’t harm your search rankings. The risk isn’t harm, it’s misplaced effort: time spent on llms.txt instead of on changes with a confirmed impact on AI and search visibility.

Where should I put my llms.txt file?

At the root of your domain, so it’s reachable at yourdomain.com/llms.txt, the same convention used for robots.txt and sitemap.xml.

Should I use an llms.txt generator or build the file by hand?

Either works. A generator saves time by pulling key pages from your sitemap, which helps on larger sites. For smaller sites, building it by hand gives you tighter control over which pages get included and how each is described, and control is the whole point of the file.

The bottom line

Whether llms.txt is worth your time comes down to what kind of site you run and what you want it to achieve. Run developer docs? Add one. Run a local business chasing AI visibility? Put your effort into content and technical foundations first. Want a clear-eyed read on where your effort is best spent? Our digital marketing agency team can walk through your specific situation.