llms.txt (llm txt): what it is and how to create one

llms.txt (also written llm.txt, llm txt or llms txt) is a Markdown file served at /llms.txt on a website. It gives AI tools a short, curated guide to the site: what it is and which pages matter most. Jeremy Howard proposed it in September 2024. It is a community proposal, not an official standard. As of November 2026, Google says its Search ignores the file, and Ahrefs found that 97% of the llms.txt files it measured received no requests in May 2026. Creating one is cheap, but the benefit is unproven. This guide covers the format, setup for static sites, WordPress and Next.js, and how to check whether anything reads your file.

What is llms.txt?

llms.txt is a plain-text file written in Markdown and placed at the root of a site, for example https://www.example.com/llms.txt. Its job is to give a language model or AI agent a map of the site, so the agent does not have to infer it from menus, scripts and page clutter. Searchers spell the name several ways (llm txt, llm.txt, llms txt, llms.txt) and use them about equally. They all mean the same file. This guide uses llms.txt, the name in the proposal.

Jeremy Howard published the proposal at llmstxt.org on September 3, 2024. The project is hosted under the AnswerDotAI organization on GitHub, and the proposal page is now labeled version 2. The page says an agent uses the file on demand: when it needs information about a topic while helping a user, it reads the file and follows the links to the detail it needs. The file can sit at the site root or at a subpath such as /docs/llms.txt, where it covers the pages under that path.

llms.txt is a proposal, not a standard issued by a standards body. Rank Math's documentation describes it as a proposal rather than an official standard, and Chrome's Lighthouse documentation calls it emerging.

What goes in an llms.txt file?

The proposal defines a Markdown file with these parts, in this order:

1. An H1 with the name of the site or project. This is the only required part.
2. A blockquote with a short summary of the key information needed to understand the rest of the file.
3. Optional paragraphs or lists, without headings, with more detail.
4. Optional H2 sections that hold file lists. Each entry is a Markdown list item with a required link, [name](url), and optionally a colon followed by notes about the page.

By convention, an H2 section named Optional holds secondary links that an agent can skip when it needs a shorter context.

Here is a complete example for a made-up business, using the reserved domain example.com.

# Northwind Analytics

> Northwind Analytics is a web analytics tool for small online stores. It tracks orders, traffic sources and returning customers without cookies.

Northwind is sold by Northwind Software Ltd in Dublin. Prices are in euros and exclude VAT.

## Product

- [Features](https://www.example.com/features): what the dashboard shows and which store platforms connect
- [Pricing](https://www.example.com/pricing): plans, limits and the free trial

## Docs

- [Quickstart](https://www.example.com/docs/quickstart): install the tracking snippet and check that data arrives
- [API reference](https://www.example.com/docs/api): endpoints, authentication and rate limits

## Policies

- [Privacy policy](https://www.example.com/privacy): what data is collected and how long it is kept
- [Terms of service](https://www.example.com/terms)

## Optional

- [Changelog](https://www.example.com/changelog): release notes
- [Blog](https://www.example.com/blog): articles and case studies

These habits keep the file accurate and easy to read:

• Link to public pages that return a 200 status. Use absolute URLs.
• Write each note as a fact about what the page answers, not as a slogan.
• Keep the summary short and factual. Name the product, the audience and the one thing it does.
• Put links you would be happy for an agent to skip under Optional.
• Save the file as UTF-8 text. The proposal does not name a content type. The bot.spot file is served as text/plain; charset=utf-8.

Version 2 of the proposal adds two optional links between pages and the file. A page can point to its Markdown copy with rel="alternate" type="text/markdown", and it can point to the llms.txt file that describes it with rel="describedby". The proposal says these can be HTML <link> elements or an HTTP Link header, and that clean Markdown versions of pages can live at the same URL with .md added.

<link rel="describedby" href="/llms.txt">
<link rel="alternate" type="text/markdown" href="/docs/quickstart.md">

Treat both as optional extras. The sources reviewed for this article do not show an AI product acting on them.

Do AI systems read llms.txt?

Some tools and agents fetch the file. Google says its Search ignores it, and OpenAI, Anthropic and Perplexity do not document their crawlers reading it. As of November 2026, this is what is public.

What Google has said

Google's guide Optimizing your website for generative AI features on Google Search says you do not need llms.txt files, other AI text files, extra markup or Markdown to appear in Google Search. It says Google Search ignores them, so they neither help nor harm your visibility there, and that it is fine to create one for other services.

Google's Search team has said the same thing in public several times:

• April 2025: John Mueller compared llms.txt to the keywords meta tag, a site's own unverified claim about what it is about, according to Search Engine Journal.
• June 2025: Mueller wrote on Bluesky, "FWIW no AI system currently uses llms.txt." Search Engine Roundtable reported that he also said server logs show chatbots fetching pages but not llms.txt.
• July 2025: Search Engine Journal's report from Search Central Live in Bangkok said Gary Illyes did not see the proposal as useful for Google and had no plans to adopt it.
• June 2026: on the Search Off the Record podcast, Mueller said LLM systems cannot use a self-reported file to tell sites apart. Search Engine Journal wrote that he allowed one narrow use: helping an agent navigate a site once it is already there, such as finding purchase instructions.

Google's Chrome team treats the file differently. Lighthouse has an llms.txt audit in its Agentic Browsing category. The audit flags a server error when it requests the file and reports "not applicable" for a 404. It measures readiness for browser agents. It does not measure rankings.

What server logs show

Ahrefs published a log-based analysis on June 15, 2026. It checked 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026.

• 28% of the domains published a valid llms.txt file. Ahrefs calls this an upper bound, because its customers skew technical and SEO-aware.
• 97% of those files received no requests in May. About 1,100 domains received all the llms.txt traffic that Ahrefs measured, roughly 22,000 requests.
• Bots made 96% of the requests to files that received traffic.
• Ahrefs did not measure llms-full.txt, and it did not check whether the files followed the specification.

The table shows who made those requests, as a share of all requests to llms.txt files.

Among named bots, GPTBot was the largest AI training crawler at 4.51% and ClaudeBot made 0.8%. Search Engine Journal's summary adds that Claude Code, a coding agent, was among the top individual requesters. The retrieval bots that answer user questions in ChatGPT and Perplexity made about 1% of requests. A request shows that a file was fetched. It does not show that the content shaped an answer.

What studies say about impact

SE Ranking analyzed nearly 300,000 domains. It found llms.txt on 10.13% of them and no relationship between having the file and how often a domain was cited in AI answers. Removing the llms.txt variable from its model improved the model's accuracy, which led the authors to suggest the file may add noise rather than signal. They note that the results depend on the model and dataset.

Adoption counts depend heavily on the sample. Originality.ai tracked more than 3 million sites from June 2025 to May 2026 and counted 36,120 with an llms.txt file, up from 4,088. It counted 2,463 with an llms-full.txt file, up from 23.

Vendors that sell the feature hedge too. SEOPress, which added llms.txt support in version 9.5, says most LLMs do not rely on the file in a meaningful or standardized way and that it is not a ranking or traffic lever today.

OpenAI, Anthropic and Perplexity each publish a page on their crawlers and how robots.txt applies to them. Anthropic's page does not mention llms.txt. OpenAI's and Perplexity's pages mention it only as the index of their own documentation. None of the three describes its crawlers reading llms.txt on other sites.

Is it worth creating?

The cost is low: a short file to write, or a plugin toggle. The evidence supports a modest case.

• It plausibly helps documentation sites and developer tools, where coding agents fetch docs. The Next.js documentation publishes an index at /docs/llms.txt and points to it from its pages, and Mintlify generates one for every documentation project by default.
• It is a weak bet as a ranking or citation tactic. Google says it ignores the file, and the SE Ranking study found no link to citations.
• Treat it as one small part of being readable by AI systems. For the wider picture, see AEO vs GEO vs SEO.

If you create one, measure the outcome rather than assume it. bot.radar runs buyer questions on AI engines every day and records which brands and URLs each answer names and cites, so you can see whether your pages are cited more often after a change. Every plan includes all seven engines: ChatGPT, Google AI Overview, Google AI Mode, Gemini, Perplexity, Claude and Copilot. Plans are counted in units, where one unit is one question tracked on one engine for a month: Brand includes 150 units and Agency 450, pooled across its workspaces.

What does a real llms.txt file look like?

bot.spot publishes its own file. A shortened excerpt follows. The ... lines mark omitted text.

# bot.spot

> bot.spot is an AI visibility and optimization platform for brands and marketing agencies. Get found, cited and chosen in AI search.

...

## About this entity

bot.spot (also written Botspot) is a software company operated by Botdot Oy in Finland. It builds AI visibility and optimization software for brands and marketing agencies. It is not affiliated with Boston Dynamics or its Spot robot, with the botspot.app photogrammetry scanner, or with botspot.trade.

...

## Pages

- [Home](https://www.bot.spot/): platform overview, products, and FAQ
- [Pricing](https://www.bot.spot/pricing): plans, trial allowances and usage rates

...

## Machine-readable

- Sitemap: https://www.bot.spot/sitemap.xml
- robots.txt: https://www.bot.spot/robots.txt

The full file has these parts:

The file is generated, not hand-edited. A Next.js route builds it from the same content that feeds the on-page AI Appendix and the JSON-LD, so one copy edit updates all three. The article list comes from the CMS, and if the CMS is down the route leaves that section out instead of failing. The response is cached for an hour.

The file also falls short of the proposal in places, and a stricter version would fix them:

• It is long. The proposal's model is a short summary plus links, with detail fetched only when needed. The bot.spot file puts product descriptions and the full FAQ inline, so it reads more like a fact sheet.
• It has no Optional section, so an agent that needs a shorter context has nothing it is told to skip.
• The profile links and the Machine-readable entries are bare URLs rather than [name](url) list items.
• The line saying AI agents and crawlers are welcome is a statement of intent. Only robots.txt carries rules that crawlers can act on.

The robots.txt for bot.spot also carries a comment that points to the file. The Sitemap: directive is meant for XML sitemaps, so llms.txt cannot go there. A comment does nothing for crawlers, so it is a hint for people and tools that look for it.

How do I create an llms.txt file?

The steps are the same on any platform. Only where the file lives changes.

1. Decide what the file is for. A good test: which pages would you hand to a new colleague, or to an agent that must answer questions about your company?
2. Pick the pages. Include product and service pages, pricing, documentation, policies and your clearest explainers. Leave out thin pages, duplicates, login pages, filtered URLs and anything set to noindex.
3. Write the H1 and the blockquote. Two plain sentences: what the site is and who it is for.
4. Group the links under H2 headings and add a short note to each.
5. Move secondary links under ## Optional.
6. Publish at /llms.txt, then request the URL and read the response.

A note on generators. Plugins, documentation platforms and online tools can draft the file for you. The proposal lists Mintlify, GitBook, Yoast SEO, AIOSEO and Wix among the platforms that generate it, plus plugins for VitePress, Docusaurus and Drupal. A draft built from your sitemap mostly repeats the sitemap in Markdown, which the proposal says is not a substitute for a curated file, so edit the output down to the pages that matter.

How do I add llms.txt to a static site?

1. Create a file named llms.txt and write the content in UTF-8.
2. Put it in the folder your site generator copies unchanged into the root of the build output. In Next.js that folder is public. For other generators, check the docs for the equivalent.
3. Build and deploy.
4. Request the file with the command below and check the response.
5. Purge the CDN cache if you use one.
6. Optional: add the rel="describedby" link to your page templates, and a comment in robots.txt that points to the file.

curl -i https://www.example.com/llms.txt

You want a 200 status, a text content type and the Markdown in the body. A redirect to the home page, a 404 page that returns 200, or an HTML response means something is wrong.

How do I create llms.txt in WordPress?

As of November 2026, several WordPress plugins can generate the file. Check the settings of the one you already run before you add another.

The sources are the Yoast and Rank Math help pages, the All in One SEO guide, the SEOPress 9.5 notes and the Website LLMs.txt listing. AIOSEO's own pages differ on which plan includes llms-full.txt, so check its pricing page.

1. Open yoursite.com/llms.txt. If a file already loads, read it first. AIOSEO's documentation says the feature is on by default.
2. Choose one source for the file. If you run Yoast, Rank Math, All in One SEO or SEOPress, use its built-in feature instead of adding a second plugin.
3. Turn the feature on and save.
4. Review what the plugin includes. Remove thin pages, tag and archive pages, and anything you would not send a reader to. Rank Math leaves out noindex posts automatically. Check the others.
5. Open /llms.txt in a private window. Confirm it shows plain Markdown and lists the pages you expect.
6. Clear your caching plugin and CDN, then check again.

To write the file by hand, upload llms.txt to the WordPress root through SFTP or your host's file manager. This is the folder that contains wp-content. Then switch off any plugin feature that also writes the file.

For the page-level work that a file cannot do, see GEO and AEO on WordPress. bot.optimizer has a live, self-serve WordPress plugin. It generates an AI Appendix for each page (a summary, key facts, FAQs and technical details) and matching JSON-LD, and you review and edit the output before it publishes through your CMS. It does not list llms.txt generation among its features, so use one of the plugins above for the file.

How do I serve llms.txt from Next.js?

This section covers the App Router. There are two ways to do it.

The simplest is a static file. Put llms.txt in the public folder. Next.js serves files in public from the base URL, and it applies Cache-Control: public, max-age=0 by default. robots.txt and sitemap.xml have built-in support in the app folder, but llms.txt is not on that list, so use public or a route handler.

If you want to generate the file from data, such as pages in a CMS, add a route handler at app/llms.txt/route.ts:

// app/llms.txt/route.ts
const SITE_URL = "https://www.example.com";

const sections = [
  {
    title: "Docs",
    links: [
      { name: "Quickstart", path: "/docs/quickstart", note: "Install and send a first request" },
      { name: "API reference", path: "/docs/api", note: "Endpoints, authentication and limits" },
    ],
  },
];

// Regenerate at most once an hour.
export const revalidate = 3600;

export async function GET() {
  const body = [
    "# Example",
    "",
    "> One or two plain sentences on what Example is and who it is for.",
    "",
    ...sections.flatMap((section) => [
      `## ${section.title}`,
      "",
      ...section.links.map(
        (link) => `- [${link.name}](${SITE_URL}${link.path}): ${link.note}`,
      ),
      "",
    ]),
  ].join("\n");

  return new Response(body, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Three details matter. First, since Next.js 15 the default caching for GET route handlers changed from static to dynamic, according to the route handler docs' version history. Setting revalidate, as above, or export const dynamic = "force-static" caches the response. Second, if the data comes from a CMS, wrap the fetch in a try/catch so an outage shortens the file instead of breaking it. The bot.spot route does this. Third, check that locale redirects, authentication or other middleware do not intercept /llms.txt.

Run curl -i against the deployed URL, as in the static site steps, and confirm the status and body.

How is llms.txt different from robots.txt and sitemap.xml?

The three files do different jobs. The proposal says sitemaps list all pages for search engines while llms.txt offers a curated overview for LLMs. It says robots.txt lets automated tools know what access to a site is considered acceptable.

The sources are RFC 9309, the sitemap protocol and Google's sitemap overview. RFC 9309 states that robots.txt rules are not a form of access authorization, so they ask well-behaved crawlers to comply and do not enforce anything.

llms.txt cannot allow or block a crawler. To control AI crawlers, use robots.txt. The AI companies document separate tokens for separate jobs:

• OpenAI lists OAI-SearchBot for search, GPTBot for training and ChatGPT-User for user-initiated visits. It says robots.txt rules may not apply to ChatGPT-User.
• Anthropic lists ClaudeBot, Claude-SearchBot and Claude-User and says its bots honor robots.txt.
• Perplexity lists PerplexityBot and Perplexity-User. It says Perplexity-User generally ignores robots.txt, because a user requested the fetch.

What is llms-full.txt?

llms-full.txt is a companion file that holds the full text of your content in one Markdown document, instead of links to it. The llmstxt.org page does not define it. Documentation platforms generate it. Mintlify describes it as the whole documentation site combined into one file, with each page's full Markdown, served at /llms-full.txt next to /llms.txt. All in One SEO offers it as a separate toggle.

The two files do different jobs:

A full export can run past what an assistant can load at once, so keep the short llms.txt as the entry point and treat llms-full.txt as an extra. Adoption is small. Originality.ai counted 2,463 sites with an llms-full.txt file in May 2026, and Ahrefs did not measure requests for it. If your documentation changes often, generate the file from the same source as your pages, as Mintlify does, rather than pasting text by hand.

How can I check whether AI crawlers fetch my llms.txt?

Start with the file, then look at the logs.

1. Confirm the file is reachable. Run curl -i https://www.example.com/llms.txt and check for a 200 status, a text content type and Markdown in the body.
2. Open your raw access logs from your web server, CDN or host. Analytics scripts usually miss crawlers that do not run JavaScript, so they are the wrong place to look.
3. Search the logs for requests to the path, as shown below.
4. Match the user agents against the table that follows.
5. Check the status codes. Anything other than 200 means crawlers are not getting the file.

On nginx or Apache with the standard combined log format, this prints a count of requests per user agent:

grep ' /llms.txt ' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -20

This keeps only the AI bots listed in the table:

grep ' /llms.txt ' access.log | grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User'

This prints how many requests returned each status code:

awk '$7 == "/llms.txt" {print $9}' access.log | sort | uniq -c

The user agents to look for, from each company's documentation:

A user-agent string can be faked. Compare the source IP address with the lists the companies publish: OpenAI's are linked from its bots page, Anthropic's is at claude.com/crawling/bots.json, and Perplexity publishes one for each bot.

Read the results with care:

• Many requests will come from SEO audit tools, profilers and research crawlers. In the Ahrefs data, those outnumbered AI retrieval bots by a wide margin.
• Seeing no requests at all is common. Ahrefs found that 97% of files received none in May 2026.
• A fetch is not a citation. To learn whether the file changes what AI systems say about you, track the answers themselves.
• If your host gives you no raw logs, look for bot or edge analytics in your CDN settings.

bot.mcp can also check the file from inside ChatGPT or Claude. Its site audit is a homepage snapshot that checks crawler access for OAI-SearchBot, Claude-SearchBot and PerplexityBot, plus robots.txt, sitemap and llms.txt.

What are the most common llms.txt mistakes?

• Treating it as access control. llms.txt cannot allow or block a crawler. Use robots.txt for that.
• Expecting rankings. Google says it ignores the file, and the studies found no link to AI citations.
• Serving the wrong thing. A redirect to the home page, an HTML error page that returns 200, or a file behind a login breaks the file. Lighthouse flags a server error on the request.
• Putting it in the wrong place. A site-wide file belongs at /llms.txt. A file at /blog/llms.txt covers the pages under /blog/.
• Pasting the sitemap into Markdown. The point is curation. A list of every URL adds little to the sitemap you already have.
• Linking to pages that fail. Remove redirects, 404s, noindex pages and pages that need a login.
• Writing marketing copy. Mueller's criticism is that the file is self-reported. Notes that stick to facts you can verify on the page are more credible than superlatives.
• Letting it go stale. Generate it from the same source as your pages, or review it whenever you add or remove a major page.
• Running two generators. A plugin and a hand-uploaded file, or two plugins, can fight over the same URL. Pick one source.
• Blocking the bots you want to read it. A robots.txt rule that disallows an AI crawler asks it not to fetch your pages, including the llms.txt file.
• Skipping the check. Request the live URL after every deploy and cache purge.

Frequently asked questions

What is an llm txt file?

An llm txt file is usually the llms.txt file: a Markdown file at the root of a website that summarizes the site and links to its most useful pages for AI tools. Jeremy Howard proposed it in September 2024. It is a proposal and not a formal standard.

Is it llm.txt or llms.txt?

The correct name is llms.txt, with an s, as written in the proposal at llmstxt.org. People also search for llm.txt, llm txt and llms txt, and all four mean the same file. Create the file at /llms.txt.

Does llms.txt help SEO?

Google says it does not. Its guide states that Google Search ignores llms.txt files, so having one neither helps nor harms your visibility in Google Search. SE Ranking's study of nearly 300,000 domains also found no relationship between having the file and being cited in AI answers.

Do ChatGPT, Claude and Perplexity read llms.txt?

OpenAI, Anthropic and Perplexity do not document llms.txt as an input to their crawlers or answers. Server logs show GPTBot, ClaudeBot and coding agents such as Claude Code requesting the file, but the numbers are small and a fetch does not show that the content was used. Ahrefs found that retrieval bots linked to ChatGPT and Perplexity made about 1% of requests.

Is there a robots.txt for AI?

robots.txt is still the file that controls AI crawlers. OpenAI, Anthropic and Perplexity each document user-agent tokens you can allow or block, and user-initiated fetchers such as ChatGPT-User and Perplexity-User may not follow it. llms.txt does not control access.

Where do I put the llms.txt file?

Put it at the root of your domain so it loads at https://yourdomain.com/llms.txt. The proposal also allows a file at a subpath such as /docs/llms.txt, where it covers the pages under that path. It should return a 200 status and Markdown text, not a redirect or an HTML page.

Summary

llms.txt is a Markdown file at /llms.txt that gives AI tools a curated map of your site. It needs only an H1, and a good one adds a short summary, grouped links with notes and an Optional section. As of November 2026, Google says its Search ignores the file, OpenAI, Anthropic and Perplexity do not document their crawlers reading it, and Ahrefs found that 97% of files got no requests in a month. You can create one on a static site, in WordPress with an SEO plugin, or with a Next.js route, then check the response and your logs. Keep expectations modest, keep the file accurate, and measure whether AI answers cite you.