Developer & API

llms.txt

Also known as: LLMs.txt, AI crawler standard

3 min read·Updated 2026-05-06

Quick definition

llms.txt is an emerging standard for websites to publish a markdown file at /llms.txt that tells AI systems (LLMs, RAG indexers, AI search products) how to read and use the site's content — what's the canonical structure, which pages matter most, what summaries to use. Proposed by Jeremy Howard in 2024, llms.txt has gained rapid adoption across content sites.

Contents

1. What is llms.txt?
2. Why llms.txt matters for AI-driven discovery
3. Building an llms.txt file
API example
Common pitfalls
Tips
FAQ

What is llms.txt?

llms.txt is an emerging web standard, proposed by Jeremy Howard in September 2024, that defines a convention for websites to publish a markdown file at /llms.txt explaining the site's structure to AI systems. The format is intentionally simple: a markdown document with the site name, a one-paragraph overview, an organized list of key pages with summaries, and optional sections for documentation, blog posts, glossary entries, etc. AI systems (LLMs, RAG indexers, AI search products) that fetch a site can read llms.txt to understand the site's structure and key content without crawling every page.

The motivation: AI systems often hit token limits when ingesting full websites; llms.txt gives them a curated, hierarchical map of what matters. Compare to robots.txt (tells crawlers which pages to access) and sitemap.xml (lists URLs in machine-readable form for traditional search). llms.txt fills the gap for AI-specific consumption: a human-readable, content-focused summary that AI systems can ingest efficiently.

Why llms.txt matters for AI-driven discovery

Three concrete benefits. (1) AI citation accuracy — when AI search products (Perplexity, ChatGPT Search, Claude with web access) reference your site, they cite content based on what they've ingested. A clear llms.txt helps them ingest the right pages with the right summaries, reducing miscitation and increasing surface area for citation. (2) Long-tail page discovery — sites have many pages but only some matter for AI consumption. llms.txt lets you tell AI systems 'these 30 pages are the canonical reference; ignore the rest.' Improves citation quality dramatically. (3) Future-proofing — AI search is the fastest-growing discovery surface; llms.txt is the de facto standard for AI-readable site structure. Sites that adopt early get earlier integration with AI products as they mature their llms.txt support.

Adoption has been rapid since the September 2024 proposal. Anthropic, Cursor, OpenAI's documentation, Vercel, Cloudflare, and many other tech-forward sites publish llms.txt files. CodivUpload publishes both llms.txt (concise) and llms-full.txt (full content dump for AI training context).

Building an llms.txt file

Five sections that most well-formed llms.txt files include. (1) Site title — H1 heading with the site name. (2) Project description — 1-2 paragraph overview of what the site does, who it serves. (3) Key links — flat list of canonical URLs (homepage, pricing, docs, glossary). (4) Optional sections — H2-organized lists of subsections (Documentation, Blog, Glossary, API Reference). Each subsection is a markdown list of links with one-line summaries. (5) Optional 'Optional' section — pages that are useful but not core; AI systems can deprioritize.

A companion file, llms-full.txt, provides the full content of all pages concatenated as one large markdown document. Useful for AI systems to ingest the site's full content in one fetch rather than crawling every URL. CodivUpload publishes both.

Minimal llms.txt example

json

# CodivUpload

> Multi-platform social media scheduling and live streaming SaaS.
> 11 platforms (Instagram, TikTok, X, YouTube, LinkedIn, etc.) with REST API,
> MCP server, SDKs, and AI-friendly tooling.

## Documentation

- [API documentation](https://codivupload.com/docs): REST API reference + Scalar UI
- [Glossary](https://codivupload.com/glossary): 130+ social media term definitions
- [Pricing](https://codivupload.com/pricing): Plans, limits, and feature matrix

## Blog

- [Best time to post on Instagram](https://codivupload.com/blog/best-time-instagram)
- [How TikTok's algorithm works](https://codivupload.com/blog/tiktok-algorithm)

## Optional

- [Changelog](https://codivupload.com/changelog): Product release history

Common pitfalls

×Listing every page on the site — defeats the curation purpose; AI systems need filtered key pages
×Forgetting to update llms.txt when site structure changes — outdated structure misleads AI
×Skipping summaries on link list — AI systems use summaries to decide which pages to fetch
×Building llms.txt with no llms-full.txt counterpart — limits AI ingestion options
×Overloading with marketing copy instead of structural information — purpose is signal, not promotion

Tips

✓Keep llms.txt short and curated — 30-50 key links is more useful than 1000
✓Include one-line summaries on every link — AI systems use them for selection
✓Update llms.txt monthly or when major content launches happen
✓Publish both llms.txt (curated) and llms-full.txt (full content dump)
✓Treat the file as you'd treat a sitemap — keep it accurate, current, and well-structured

Frequently asked questions

Do AI systems actually use llms.txt?+

Increasingly yes. Anthropic Claude, Perplexity, and Cursor are known to consume llms.txt when available. ChatGPT Search and Bing Copilot are testing support. Adoption is growing rapidly through 2026.

Is llms.txt different from robots.txt?+

Yes. robots.txt tells crawlers which URLs to access (gating). llms.txt tells AI systems how to interpret the site's content (semantics). Both are useful; they serve different purposes.

Should I publish both llms.txt and llms-full.txt?+

Yes — they serve different consumption modes. llms.txt is a curated index; llms-full.txt is full content for AI ingestion. Best practice is to publish both.

Where does llms.txt live?+

At the root of your domain: /llms.txt (and /llms-full.txt for the full content version). Same convention as robots.txt and sitemap.xml.

Will llms.txt help my SEO rankings?+

Indirectly. It doesn't affect traditional SEO ranking position. It improves AI citation accuracy and AI-product visibility, which feeds into longer-term brand-impression value. Worth publishing regardless.