Quick answer: What is llms.txt for API documentation?

llms.txt is a small Markdown file that gives AI agents a curated map of a website and points them to its most useful content. On a documentation site, it helps agents find authoritative content on the right pages without having to scan the entire site. It is an emerging, optional convention, so teams should not assume every AI tool will discover or use it automatically.

When creating an llms.txt file, focus on four areas:

• Goal: Explain the site and route agents to authoritative content.

• Sections: Start with an H1 title, then optionally add a summary, explanatory context, grouped H2 link lists, and an Optional section for secondary links.

• Format: Use Markdown links with clear labels and concise descriptions.

• Delivery: Serve it at /llms.txt or a scoped path and link to clean Markdown versions of important pages.

Below, we explore how to structure, organize, deliver, maintain, and test llms.txt for API documentation, as well as how Docuwiz can support the underlying documentation workflow.

What is llms.txt?

llms.txt is an emerging convention for publishing a Markdown guide that helps AI agents understand a website and find its most useful content. It gives agents brief context and curated links to authoritative resources, helping them find the right pages without scanning the entire website.

An llms.txt file requires only an H1 naming the site or project. Teams can then add a short blockquote summary, explanatory notes, and H2 sections that group relevant links, with an optional description for each. By convention, secondary resources go under an Optional heading so agents can leave them out when a shorter context is sufficient.

This is not a mandatory web standard. Chrome's Lighthouse documentation describes llms.txt as an emerging, optional convention, so websites are not required to publish one. Publishing llms.txt creates a retrieval path, but does not guarantee discovery, ranking, citation, or correct use by every AI system.

The convention was revised in August 2026. Version 2 is the first update since the original 2024 proposal, and it changes two things that matter for documentation teams: it permits a second URL pattern for Markdown versions of pages, and it introduces standard link relations so a page can point to both its Markdown twin and the llms.txt file that describes it. Both are covered later in this guide.

Teams can publish the file at the site root, such as https://example.com/llms.txt, or at a scoped path, such as https://example.com/docs/llms.txt. A file published under a path describes the content within that part of the website.

Is publishing llms.txt worth the effort? 

Two facts should shape that decision before anything else in this guide.

It will not help your search rankings. In June 2026, Google updated its AI optimization guidance to state that llms.txt has no effect — positive or negative — on Search rankings or AI Overviews. Any guide presenting this file as an SEO technique is selling something. The reason to publish one is agent retrieval, and that is a different question from search visibility.

Most published files are invalid. A survey of top sites in mid-2026 found that of 636 scorable pages, 186 had attempted an llms.txt file and only 64 had a valid one. Roughly two-thirds of attempts were malformed. A broken file is worse than none, because it signals a retrieval path that does not work.

That leads to a straightforward test. Publish one if your documentation serves developers whose tools retrieve it, if your site is large enough that an agent would otherwise have to guess which page is authoritative, or if you maintain multiple products or API versions on the same domain. Skip it if your documentation is a handful of pages an agent can read in full, or if nobody will own keeping it current — an unmaintained file pointing at moved pages does active harm.

Why does llms.txt matter for API documentation?

API documentation is rarely contained on a single page. An agent may need a quickstart to understand the first request, an endpoint reference for parameters and schemas, an authentication guide for access requirements, and separate guidance for workflows, errors, or webhooks. Finding the right combination becomes more difficult when multiple products, versions, or environments share the same documentation site.

llms.txt can provide a structured starting point. It can route agents to the relevant guide, reference, or product-specific index and explain what each resource contains. This helps agents choose sources that match their task instead of searching broadly across the documentation site.

However, llms.txt only improves discovery, not the underlying content. It cannot fix incomplete endpoint descriptions, inaccurate examples, missing permission requirements, or outdated reference material. The linked pages must still provide the precise schemas, workflow steps, side effects, error-recovery guidance, and version context required for agent-ready API documentation.

How do AI agents use llms.txt to find the right API documentation?

An AI agent can retrieve or search llms.txt, read its summary and guidance, and use the link descriptions to decide which resources match its task. For a webhook integration, for example, an agent can use the link titles and descriptions to identify the relevant authentication guidance, webhook guide, and API reference. It can then retrieve those pages for details such as event schemas, signature verification, retries, and troubleshooting.

This is the intended usage pattern, not behavior that every AI tool follows automatically. Some tools may discover llms.txt on their own, while others access it only when a user, integration, or retrieval system directs them to the file.

A website can publish either a single llms.txt file or multiple files at different paths. A root file at /llms.txt can cover the entire website, while a path-specific file such as /docs/llms.txt covers the content within that section. When more than one file applies, the agent uses the most specific one. This allows larger documentation sites to provide a high-level directory alongside narrower indexes for individual products or documentation sets.

After locating the applicable file, a typical agent retrieval process looks like this:

1. The agent reviews the summary and any guidance about versions, environments, or authoritative sources.

2. It selects the links that best match the task.

3. It retrieves the linked Markdown pages or other machine-readable resources.

4. It confirms that the information applies to the correct API version and environment, then checks authentication requirements, side effects, and error behavior before relying on the content.

The OpenAI developer documentation index demonstrates this hierarchical approach. Its root file groups resources into documentation sets and common tasks, then routes agents to narrower indexes for areas such as API guides and endpoint references. This allows an agent to narrow its search before retrieving detailed pages.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt, sitemap.xml, and llms.txt can all help automated systems interact with a website, but they serve different stages of the process: crawl control, URL discovery, and content selection.

RFC 9309 defines robots.txt as the Robots Exclusion Protocol. It communicates crawling preferences; it does not protect private content or grant access to restricted documentation. Sensitive API documentation still requires authentication and authorization.

The Sitemap protocol supports URL discovery. An API documentation sitemap can list endpoint pages, guides, changelog entries, and other published URLs, but it does not explain which resources are authoritative or which combination an agent needs for a specific task.

llms.txt is the layer that adds context to URL discovery. It can direct an agent to the current quickstart, authentication guide, reference index, error documentation, and version policy without listing every URL on the site. It complements robots.txt and sitemap.xml; it does not replace either one.

Each file has a distinct role: robots.txt provides crawl guidance, sitemap.xml supports URL discovery, and llms.txt supports task-oriented resource selection. Teams need to keep all three aligned with the public documentation.

How does llms.txt relate to MCP?

Teams building agent-facing documentation often ask whether llms.txt and the Model Context Protocol solve the same problem. They do not, and a team may need both.

llms.txt is pull-based discovery. An agent that has found your documentation — through a search result, a link, or a user instruction — uses llms.txt to decide which pages to retrieve. Nothing has to be connected in advance, and you have no relationship with the agent or its operator.

MCP is a connected tool interface. An agent configured with an MCP server calls defined tools rather than retrieving and parsing pages. The connection is deliberate, the available operations are explicit, and access can be governed.

The practical distinction is who initiates and what you control. llms.txt improves the odds that an agent reading your public documentation reaches the right page; it offers no authentication, no access control and no usage visibility. An MCP server gives you all three, for agents you have agreed to serve.

Most documentation teams will want llms.txt first, because it is cheap and covers the unbounded case of agents arriving from anywhere. Teams exposing APIs to partner agents, or to agents acting inside their own products, will eventually want the connected interface as well. The two layers also share a dependency: both are only as good as the underlying content, and neither compensates for documentation that omits permissions, side effects or error behavior.

What should an API documentation llms.txt file contain?

An API documentation llms.txt file should give an agent enough context to choose the right source without trying to reproduce the documentation itself. The format is intentionally lightweight. It begins with an H1 naming the site or project, which is the only required element. Teams can then add a short blockquote summary, a few lines explaining the file's scope or identifying authoritative sources, and H2 sections that group relevant links with clear descriptions.

Section names are flexible and should reflect how an agent navigates the documentation. A larger documentation site might separate broader documentation sets from common tasks, while a smaller site might link directly to its most important pages. Here is an example:

# Acme Payments API

> Guides and reference documentation for integrating Acme Payments API v2.

Start with the quickstart for your first sandbox request.
Use the API reference for endpoint parameters and schemas.

## Documentation sets

- [API guides](https://docs.example.com/guides/llms.txt): Concepts and implementation workflows for Acme Payments.
- [API reference](https://docs.example.com/reference/llms.txt): Endpoints, request parameters, response schemas, and authentication requirements.

## Common tasks

- [Quickstart](https://docs.example.com/quickstart.md): Create credentials and make a sandbox payment.
- [Authentication](https://docs.example.com/authentication.md): Configure API keys, OAuth scopes, and environment access.
- [Create and capture a payment](https://docs.example.com/guides/payments.md): Follow the payment workflow from creation through capture.
- [Verify webhook signatures](https://docs.example.com/guides/webhooks.md): Validate events and handle retries or replay attempts.

## Reliability and changes

- [Errors and retries](https://docs.example.com/errors.md): Interpret error responses and determine when a request can be retried.
- [Idempotency](https://docs.example.com/idempotency.md): Prevent duplicate operations when retrying write requests.
- [Changelog](https://docs.example.com/changelog.md): Review API changes, deprecations, and release dates.

## Optional

- [SDK examples](https://github.com/example/acme-payments-examples): Sample integrations in supported programming languages.

Use absolute, canonical URLs where possible. Describe each link by what the agent will find there rather than using labels such as Learn more. A smaller documentation site can link directly to its key pages, while a larger site can route agents to narrower, path-specific llms.txt indexes. In either case, the detailed information remains on the linked pages; the llms.txt file provides the context needed to select them.

Use Markdown link syntax for every entry, including entries that are already plain URLs. Chrome's Lighthouse audit fails files whose links are not formatted as Markdown links, even when every link resolves correctly — a common reason an otherwise accurate file is reported as broken.

How should references, guides, authentication, and examples be organized?

Organize the links around the information an agent needs to select, not the documentation team's internal folder structure.

• Provide orientation and prerequisites first: State which API and version the file covers, identify the authoritative source when necessary, and clarify relevant environments. Make authentication methods, permissions, scopes, and credential requirements easy to find so an agent can confirm access requirements before using the API.

• Separate contracts from workflows: Group the OpenAPI description, endpoint reference, schemas, response formats, and error definitions as reference material. Group multi-step procedures, webhook setup, migrations, and troubleshooting as guides or workflows. This helps an agent distinguish between looking up an exact detail and learning how to complete a task.

• Describe examples by task: When linking to examples, use titles and descriptions that explain what each example demonstrates. Include the language, SDK, API version, or environment when that context affects how the example should be used.

• Reserve `Optional` for secondary resources: Use this section for links that an agent can skip when shorter context is needed. Whether a resource is optional depends on the scope of the file and the tasks it supports, not simply on the type of content.

How do you provide Markdown-accessible documentation pages?

The current llms.txt v2 proposal recommends providing clean Markdown alternatives for pages that agents may need to retrieve. For example, the Markdown version of /docs/authentication.html could be served at /docs/authentication.md or /docs/authentication.html.md. For a URL that represents a directory, the proposal recommends appending index.md or index.html.md.

The relationship between the original page, its Markdown alternative, and the applicable llms.txt file can be exposed through HTML <link> elements or HTTP Link headers. An HTTP response can advertise both resources like this:

Link: </docs/authentication.md>; rel="alternate"; type="text/markdown",
      </docs/llms.txt>; rel="describedby"

Generate the HTML and Markdown versions from the same content source where possible. Preserve headings, tables, lists, code blocks, warnings, and links in the Markdown output, while removing navigation, banners, decorative labels, and repeated footer content.

Prioritize Markdown alternatives for the pages listed in llms.txt, such as quickstarts, authentication guidance, API references, workflows, and error documentation. When a page moves or changes version, update its HTML output, Markdown output, and llms.txt entry together. This applies the same source-of-truth discipline used for keeping API documentation in sync.

What llms.txt implementation mistakes should teams avoid?

• Treating `llms.txt` as a ranking switch: Publishing the file does not guarantee higher search rankings, AI citations, or preferred inclusion. Test whether the AI tools relevant to your audience discover and use the file as intended.

• Using plain URLs instead of Markdown links: Lighthouse fails files whose entries are not Markdown-formatted links, regardless of whether the links work. This is one of the most common reasons a correct-looking file is reported as broken.

• Copying the sitemap: Do not turn llms.txt into an unexplained list of every URL. Select authoritative task paths and describe why each link matters.

• Linking only to rendered HTML: Agents can read HTML, but heavily scripted pages and repeated interface text make extraction harder. Provide clean Markdown alternatives for frequently retrieved pages.

• Publishing stale or conflicting links: Generate the file from current navigation or content metadata where possible, and include it in documentation release reviews.

• Letting the file return a server error: Lighthouse marks a missing file as not applicable, which carries no penalty, but a server error when fetching it fails the audit. A file that intermittently 500s is worse than no file at all.

• Exposing private material: Do not list internal specifications, customer-only guides, unpublished endpoints, test credentials, or private repository URLs in a public file. Protect restricted resources with authentication and authorization.

• Using vague labels: Labels such as Docs, API, and Learn more provide little basis for selection. Name the task, API area, version, or constraint.

• Skipping retrieval tests: A file can appear correct but still direct agents to the wrong source, version, or insufficient content. Test realistic questions and confirm that the agent selects the appropriate pages and retrieves enough context to answer accurately.

How do you know whether it is working? 

The file itself produces no metrics, so judge it on retrieval behavior and maintenance cost rather than on publication.

Validity. Does the file pass the Lighthouse agentic browsing audit, resolve without server errors, and contain only live links? Check this on a schedule, not once.

Retrieval accuracy. Give an agent a set of realistic tasks — make a first request, configure OAuth, handle a rate limit, verify a webhook signature, migrate a deprecated endpoint — and record whether it selects the correct pages and the correct API version. Repeat the same task set after each documentation release; a drop here usually means a link moved or a new page should have been listed.

Sufficiency. When the agent selects the right pages, does it still miss permissions, side effects, or error recovery? That is a content problem rather than an llms.txt problem, and it tells you where the linked pages need work.

Referral evidence, where available. Some documentation platforms and CDNs can report requests for /llms.txt and for .md alternatives separately from HTML traffic. That will not tell you which agent is retrieving them or why, but a flat line after several months is a reasonable signal that the audience you expected is not arriving this way.

Maintenance effort. Track how long the file stays accurate after each release. If it needs manual correction every time, generate it from navigation or content metadata instead.

How does Docuwiz support an AI-readable documentation workflow?

A reliable llms.txt implementation depends on structured, accurate, and well-maintained documentation. Docuwiz supports this foundation by bringing API references, guides, recipes, revisions, and publishing controls into the same documentation workspace.

Against the steps in the checklist below:

Defining scope and curating links. Teams can import .yaml, .yml, and .json API descriptions to generate structured reference pages, and create or import Markdown guides in visual or Markdown editing modes. Recipes combine explanatory pages and API operations into task-oriented documentation, which maps more closely to how an agent selects a resource than an endpoint reference alone does.

Serving accessible content. Published portal URLs become the targets listed in llms.txt, and because references are generated from the API description, the reference pages stay aligned with the contract rather than drifting from it.

Maintaining the file. GitHub and GitLab source-control workflows, document comments, revision states, and publish or unpublish controls let teams review which version appears in the published portal — and review the corresponding llms.txt entries in the same release process rather than as a separate task.

Once the relevant guides, references, and recipes are published, teams can add their public portal URLs to llms.txt and keep both under the same review cycle.

Teams can import .yaml, .yml, and .json API descriptions to generate structured reference pages. Writers can create or import Markdown guides and work in visual or Markdown editing modes. Recipes can combine explanatory pages and API operations into task-oriented documentation that provides more context than an endpoint reference alone.

Docuwiz also supports GitHub and GitLab source-control workflows, document comments, revision states, and publish or unpublish controls. These capabilities help teams review documentation and determine which version should appear in the published portal.

Once the relevant guides, references, and recipes are published, teams can add their public portal URLs to llms.txt. When the documentation changes, the pages and corresponding llms.txt links can be reviewed as part of the same release process.

llms.txt implementation checklist

☐ Define the scope: Decide whether the file covers the whole site, a documentation path, a product, or an API version.

☐ Add essential context: Name the site or project and clarify relevant versions, environments, and authoritative sources.

☐ Curate the links: Select the most important guides and references and give each one a clear title and description.

☐   Format every entry as a Markdown link: Plain URLs fail validation even when they resolve.

☐ Serve accessible content: Link to clean, current, publicly accessible pages and provide Markdown alternatives where possible.

☐   Advertise the relationships: Add rel="alternate" and rel="describedby" link relations, in HTML or as an HTTP Link header.

☐ Test retrieval: Use realistic tasks to confirm that an agent selects the correct sources and versions.

☐ Maintain the file: Assign an owner and review it whenever documentation, navigation, or API versions change.

Key takeaways

Use llms.txt as a curated starting point for AI agents, not as a replacement for OpenAPI, robots.txt, sitemap.xml, access controls, or accurate documentation. Do not assume that publishing it guarantees discovery, ranking, or citation.

Start with an H1, then add a concise summary, essential version or environment guidance, and grouped links with clear descriptions. Publish the file at the root or relevant path and direct agents to clean, current, and publicly accessible resources.Follow version 2 on both counts that matter: accept either Markdown URL pattern, and advertise the Markdown alternative and the describing llms.txt through link relations.

Treat llms.txt and MCP as different layers. The first helps an agent that found you choose the right page; the second gives agents you have agreed to serve a governed interface. Neither compensates for documentation that omits permissions, side effects, or error behavior.

Test realistic retrieval tasks to confirm that agents select the correct documentation and receive enough context to use the API accurately. Update the file alongside documentation releases, and use Docuwiz to maintain the structured references, guides, revisions, and published pages that support it.

FAQs

Is llms.txt an official web standard?

No. It is an open, emerging convention rather than an IETF or W3C standard. Chrome Lighthouse can audit it, but Chrome's documentation says the file is optional. Treat it as a retrieval aid and verify behavior in the AI tools your users rely on.

Does llms.txt improve AI rankings or citations?

There is no specification-level guarantee that publishing the file improves ranking, citation, or inclusion in an AI answer. It can make authoritative resources easier for a compatible agent to find. Content quality, accessibility, freshness, and the consumer's retrieval behavior still determine whether the material is used correctly.

Do we need llms.txt if we publish an MCP server?

They cover different situations. An MCP server serves agents you have deliberately connected, with defined tools and governed access. llms.txt helps an agent that arrived on its own choose the right page from your public documentation. Teams with public docs and partner integrations usually end up wanting both.

Should llms.txt include every API endpoint?

Usually not as individual links. Point to the current OpenAPI description or reference index, then highlight the most important quickstarts, workflows, auth rules, errors, and operational guidance. Very large documentation sets can use narrower indexes by product, path, or version.

Is llms.txt a replacement for OpenAPI?

No. OpenAPI describes the API contract: operations, parameters, schemas, responses, and security schemes. llms.txt helps an agent discover that contract and the guides that explain how to use it.

How often should llms.txt be updated?

Update it whenever an authoritative page moves, a supported API version changes, a new core workflow ships, or a linked resource is retired. Generating it from shared navigation or content metadata is safer than maintaining a separate manual list.

How should a team test llms.txt?

Check syntax and links first. Then give an agent realistic tasks such as making a first request, configuring OAuth, handling a rate limit, verifying a webhook, or migrating an endpoint. Confirm that it selects the correct version and sources and does not miss permissions, side effects, or recovery guidance.

Related reads

• Agent-ready API documentation

• How to keep API documentation in sync

• API documentation examples and patterns