- What is an llms.txt file and why does my website need one?
- llms.txt is a standardized Markdown file placed in your root directory that provides AI search engines and LLM agents with a curated, concise guide to your website content.
- Where should I host the llms.txt and llms-full.txt files?
- Host them at the root of your web domain: https://example.com/llms.txt and https://example.com/llms-full.txt, just like robots.txt.
- How does llms.txt improve Generative Engine Optimization (GEO)?
- It removes HTML overhead, navigation noise, and scripts, providing AI search engines with clean token representations that are easily cited in answers.
- What is the difference between llms.txt and llms-full.txt?
- llms.txt contains a curated index of links and summaries. llms-full.txt aggregates the full markdown text of all documentation pages into a single file for comprehensive agent context.
- What is llms.txt and what problem does it solve?
- llms.txt is an emerging open standard (llmstxt.org) designed to provide a lightweight, curated Markdown guide to a website's most useful resources for Large Language Model (LLM) agents and AI search systems. While robots.txt manages crawler access permissions and sitemap.xml dumps every URL on a domain, llms.txt provides a clean human- and machine-readable table of contents highlighting high-value documentation, product pages, and developer guides.
- What is the current llms.txt v2 format specification?
- Under the v2 specification, an llms.txt file begins with a single required H1 title (# Project Name), followed immediately by an optional blockquote summary (> Short summary of the project), optional explanatory markdown, and zero or more H2 sections (## Section Name). Each section contains a Markdown bullet list of links following the format: - [Title](URL): Optional description. Secondary or auxiliary resources may conventionally be placed under "## Optional".
- What is the difference between llms.txt and robots.txt?
- robots.txt and llms.txt solve entirely different problems and neither overrides the other. robots.txt specifies technical crawl permissions (Allow/Disallow) for search engines and web scrapers. llms.txt provides content orientation, telling AI agents which URLs are most authoritative, what each page covers, and where clean markdown documentation can be fetched.
- What is the difference between llms.txt and sitemap.xml?
- sitemap.xml is an exhaustive XML catalog intended to list all indexable URLs on a website for search engines. Dumping thousands of sitemap URLs into an llms.txt file is considered an anti-pattern because LLMs have finite context windows. llms.txt is intended to be a curated index of your 10 to 50 most essential resources.
- Does llms.txt only belong at the domain root, or can it exist at subpaths?
- llms.txt can exist both at the domain root (https://example.com/llms.txt) and at path levels (e.g. https://example.com/docs/llms.txt or https://example.com/api/llms.txt). A path-level file applies to content beneath that specific subpath. If multiple llms.txt files apply, AI agents treat the most specific path-level file as the authoritative guide for that section.
- How do AI agents discover llms.txt on a website?
- AI agents discover llms.txt in three primary ways: 1) by probing the standard root path /llms.txt; 2) via HTML <link rel="describedby" href="/llms.txt"> tags on the homepage; and 3) via HTTP Link headers (Link: </llms.txt>; rel="describedby").