At a Glance
- llms.txt is not a magic AI-visibility unlock, it’s a curated, machine-readable guide that helps LLMs understand your most important pages.
- Adoption exists (Perplexity, Anthropic, Cursor, Stripe, Cloudflare), but no major answer engine has announced official support and server logs show inconsistent crawling.
- It’s useful only for specific cases, especially API-driven products or developer ecosystems and counterproductive when misused as SEO/AEO “growth hack.”
Why This Question Even Exists
LLMs struggle with websites. Without grounding (real-time data access), ChatGPT, Claude, and others often can’t tell:
- which pages matter most
- what task flows exist
- how your information architecture is structured
- which content is authoritative vs. secondary
This gap created space for community-driven solutions and llms.txt emerged as one of them. But as with every new standard, implementation has ranged from thoughtful to reckless. Some teams add a well-crafted llms.txt. Others create .md clones of every blog post (a bad idea). Some ask whether Google punishes this. Others claim “you must do this for AEO.” Let’s break down what’s actually happening.
What Is llms.txt?
llms.txt is a simple, markdown-formatted text file placed at the root of your domain that gives LLMs a curated overview of your most important content. Think of it as a helpful map for AI agents, not a search-ranking tool. The protocol includes two files:
1. llms.txt
A short list of high-value pages — typically:
- homepage
- docs or knowledge base
- pricing
- support
- terms
- product/service overview
- FAQ pages
2. llms-full.txt
A long-form, detailed companion containing:
- walkthroughs
- examples
- workflows
- API usage
- troubleshooting
- step-by-step instructions
Some llms-full.txt files exceed 30,000+ words.
Example Structure (Perplexity)
Perplexity’s llms.txt uses headings, short descriptions, and markdown links:
# Perplexity
## Docs
- [List Async Chat Completions](https://docs.perplexity.ai/api-reference/async-chat-completions-get.md)
- [Pricing](https://docs.perplexity.ai/getting-started/pricing.md)
- [System Status](https://docs.perplexity.ai/status/status.md)
## Optional
- [Forum](https://community.perplexity.ai)
- [Blog](https://perplexity.ai/blog)
- [Changelog](https://docs.perplexity.ai/changelog/changelog)
Their llms-full.txt is over 7,000 lines and includes executable examples, such as setup instructions for API keys.
Where Did llms.txt Come From?
The protocol was introduced in September 2024 by Jeremy Howard — founder of FastAI and AnswerAI, and lecturer at the University of Queensland and Stanford. It gained early traction among AI-first companies.
As of July 2025, 600+ websites use it, including: Perplexity, Anthropic, Cursor, Stripe, Solana, ElevenLabs, Hugging Face, Raycast, Yoast, DataForSEO, Zapier, Cloudflare. This is real adoption but not mainstream crawler infrastructure.
How Does It Compare to robots.txt and sitemap.xml?
llms.txt vs robots.txt
The llms.txt file differs entirely from robots.txt in purpose. Robots.txt is used to control crawler access by specifying which paths are allowed or disallowed, while llms.txt contains no access rules at all and simply provides helpful context for language models. This means robots.txt remains the true control point for managing LLM or any other crawler behavior, whereas llms.txt is purely informational.
llms.txt vs sitemap.xml
Compared with sitemap.xml, llms.txt is much more selective. A sitemap lists every important URL on a site so that search engines can discover its full structure, but llms.txt is intentionally short and curated, highlighting only the pages that matter most for understanding or assisting users. For example, a large retailer might have millions of product URLs in its sitemap, while its llms.txt would focus on essentials such as the return policy, FAQ, and checkout overview, with an optional llms-full.txt providing deeper guidance on tasks like search, filtering, checkout steps, or API usage.
What Does llms.txt Look Like in Practice?
Perplexity’s llms.txt is structured as such:
# Perplexity
## Docs
- [List Async Chat Completions](https://docs.perplexity.ai/api-reference/async-chat-completions-get.md): Lists all asynchronous chat completion requests for the authenticated user.
[...]
- [API Groups & Billing](https://docs.perplexity.ai/getting-started/api-groups.md): Learn how to use the Perplexity API Portal to manage access, usage, billing, and team collaboration.
- [Pricing](https://docs.perplexity.ai/getting-started/pricing.md)
[...]
- [System Status](https://docs.perplexity.ai/status/status.md)
## Optional
- [Forum](https://community.perplexity.ai)
- [Blog](https://perplexity.ai/blog)
- [Changelog](https://docs.perplexity.ai/changelog/changelog)
Their llms-full.txt includes - examples for API calls; environment variable setup; troubleshooting notes.
[...]
## 🔑 API Key Setup
All examples require a Perplexity API key. You can set it up in several ways:
### Environment Variable (Recommended)
```bash
export PPLX_API_KEY="your-api-key-here"
```
### .env File
Create a `.env` file in the example directory:
```bash
PERPLEXITY_API_KEY=your-api-key-here
```
[...]SEOs & llms.txt: Where Things Went Wrong
Many SEOs tools / experts have begun adding links to every landing page, generating markdown copies of all their blog posts, and filling llms.txt files with massive lists of URLs. These practices often create unnecessary complications such as duplicate content across the site, low-value markdown clones that serve no real purpose, wasted crawl budget as search engines process redundant pages, and diluted signals that make it harder for algorithms to interpret what truly matters. Despite the effort, these tactics deliver no clear or measurable advantage for answer-engine optimisation.
John Mueller’s View
John Mueller is a Senior Webmaster Trends Analyst at Google LLC. As part of Google’s Search Relations team, he engages directly with webmasters and SEOs, helping clarify how Google Search evaluates and processes websites. His insights are particularly valuable because they reflect Google’s internal perspective on crawling, indexing, and web content quality.
When asked about sites hosting markdown clones, Mueller responded that:
AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag - this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)”
This statement offers one of the most explicit warnings from Google against using an llms.txt file as an SEO tactic.
llms.txt & Answer Engines
As of November 2025, no major LLM-powered search engine has confirmed support for llms.txt, server logs rarely show meaningful crawler activity, and there’s no proven impact on answer-engine optimisation or AI Overviews. That means llms.txt is not an SEO tool. Its real value appears in technical contexts such as SaaS platforms, developer tools, and products that rely on agents or automation, where it serves as helpful documentation. For content-driven marketing sites, though, llms.txt offers little benefit and often just creates additional noise.
Getting Started With llms.txt (If It Fits Your Use Case)
Step 1: Create llms.txt
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)
You can use the code samples above or check the official llmstxt.org reference for guidance on recommended formats. This directory also includes examples from different companies that show how they organise their llms.txt structure in practice.
Step 2: Host at your root
Various website builder now support for hosting of llms.txt. If you are on Webflow, you can now do this by or refer to this exhaustive guide:
- Go to your Site settings > SEO > LLMs.txt
- Click Upload file
- Choose your llms.txt file (must be UTF-8 encoded and under 100 KB)
- Click Save changes
After publishing your site, the file will be available at https://example.com/llms.txt. Note that it won’t be published to your Webflow staging domain.
Step 3: Test It
- Test 1: Retrieval: Ask ChatGPT or Claude to fetch it. If they don’t retrieve it → no usage.
- Test 2: Local Chat: Paste the file into a fresh chat with grounding disabled. See if the LLM can answer questions based solely on your file.
- Test 3: Agent Performance (Optional): Give an agent a task (e.g., “Find our refund policy and summarise next steps”). Then repeat with llms.txt included as system context. Measure improvement.
- Test 4: Vibe Coding Tools: Provide your files to Cursor or Claude Code. Check whether prototypes interact correctly with your API or structure.
Underscore’s Perspective for Webflow & B2B Teams
For most growth-stage B2B companies and mid-market service firms, llms.txt isn’t a meaningful priority. AI visibility is driven by fundamentals such as strong semantic structure, well-built CMS architecture, clear internal links, schema markup, fast performance, and authoritative content. These are the core levers of AEO. llms.txt only becomes strategically important for developer-focused products, API-based SaaS platforms, or tools designed for AI agent usage. For everyone else, it’s optional and can even be a distraction.
Our Recommendations
- Fix your architecture first. AEO rewards structure, not tactical add-ons.
- Use llms.txt only if agents/developers will consume it. Otherwise, invest in foundational visibility
- Avoid markdown clones or link dumps. They create duplicate content and harm clarity.
Conclusion & Next Step
llms.txt is a credible idea with niche usefulness but it is not a visibility hack, nor a replacement for real SEO/AEO foundations. For 95% of B2B and mid-market teams, the priority should be: architecture, clarity, schema, and performance.
If you want to understand whether llms.txt adds value for your site, our Webflow AEO audit can map that out clearly.
