Want to create a llms.txt file for your website? Here’s everything you need to know.
There’s a new standard in town, and you’ve probably heard of it, which is why you’re here. That’s right, it’s llms.txt. In this post we’ll answer the following questions:
- What is llms.txt?
- How is llms-full.txt different from llms.txt?
- What are the elements of llms.txt?
- How does llms.txt fit into an AI SEO strategy?
- How do I create a llms.txt file for my website?
- What are some examples of llms.txt?
What is LLMS.TXT?
Llms.txt is the new (proposed but de facto) technical standard for large language models reading websites. It’s a plain-text markdown-based file standard designed to help LLMs (like ChatGPT, Claude, Gemini, Perplexity, and others) understand and navigate website content.
The solution to a problem: AI models struggle to parse heavy HTML and JavaScript. Llms.txt provides a roadmap of the content you identify as most valuable, making it easier for LLMs to ingest.
Sidenote: Also, you’re probably wondering why it’s “llms.txt” and not “llm.txt.” The answer is that this file is meant to adhere to a common standard used by multiple large language models, not just one. For the same reason that it’s a “robots.txt” file and not “robot,” the goal is to have as many LLMs as possible understand your site.
How is LLMS.TXT different from robots.txt
Robots.txt is a way for website owners to block (or attempt to block) robots from crawling your site. That’s not the case for LLMS.TXT, you aren’t blocking anything. Instead, you’re guiding LLMs through your content. Thing of LLMS.TXT as a map. Whereas Robots.txt is like a (not enforceable) legal notice not to trespass.
What is LLMS-FULL.TXT?
Llms-full.txt is a more complete file than llms.txt. Where llms.txt is a summary, llms-full.txt often contains more content and more detailed content than Llms.txt; sometimes the full content. This approach has some downsides, since some LLMs may have difficulty reading such a large file, and in some cases defeats the purpose of a succinct summary of a website for LLMs to navigate.
AI SEO and the LLMS.TXT standard
This is really the meat of why On Marketing is concerned with llms.txt files: because it’s a small but important tactic in getting your brand adequately cited by LLMs. Many B2B companies aren’t happy with how their products or services or thought leadership is represented by ChatGPT and others. Having a well-built llms.txt file is part of the website structure for AI that will get your brand the visibility it has earned in the marketplace.
How do I create an LLMS.TXT file?
Start by asking yourself the following questions:
Site Purpose. What is the main goal of your website in one or two sentences?
Target Audience. Who are you trying to reach with your site? What challenges or needs are they hoping to address by visiting your site?
Key Pages. Which 5 to 10 pages should be prioritized first? These might include homepage, main service or product overview, blog posts (especially your most valuable or educational content), case studies, white papers, or research, etc.
Optional Content. Are there any pages you’d like included, but given less importance?
Markdown Versions. Do you already have plain text or Markdown versions of your pages, or could you create them? (These are files ending in “.md”. A Markdown version of a page is a simplified, plain-text version of a web page written in Markdown, a lightweight markup language used to format text.
Update Frequency. How often does your site’s content or structure change?
The LLMS.TXT Structure
A llms.txt file is typically structured to organize and list various types of content from a website, such as pages, posts, and resources, and to provide details about each one. It can be divided into sections for easy reading and use, with key details like URLs, publication and modification dates, and additional metadata for each entry.
Main Structure
Sections: The file is divided into logical sections like Pages, Posts, and Resources, which categorize the types of content you’re listing.
Content Items: Each content entry contains specific details:
Title (often in the form of a URL)
Description (usually summarized in a brief paragraph or sentence)
Metadata (includes publication and modification dates, categories, or resource type)
Delimiter: — is used to separate content items or sections for clarity.
Formatting Notes
Headings: Sections are usually indicated by ##, such as ## Pages or ## Posts.
Metadata: Information like publication date, modified date, and URL are common for each entry.
Categories: Some entries have categories (e.g., SEO, AI) to group similar content.
Best Practices for LLMS.TXT files
This is an emerging space, and best practices are sure to change. But here are some best practices to start out with:
Headings. Use clear headings (#, ##), concise descriptions, bullet lists. This formatting helps large language models quickly parse and prioritize your site’s content structure without noise or ambiguity.
Impact. Limit content to the most relevant. Overloading with every URL will dilute effectiveness. Including only your most impactful pages ensures LLMs focus on content that best represents your expertise and authority.
Optional section. Create a “## Optional” section for secondary or less critical pages. A clearly marked optional section lets LLMs differentiate between primary sources and supporting or supplementary material.
Keeping up-to-date. Regularly update the file as your site evolves. If your site structure changes, update your llms.txt file. If your content or mission change, update. And regularly audit the file, since it won’t update on its own.
Real examples of LLMS.TXT
Here are 6 examples of current llm.txt files found on the web. (As of this posting, all links work. We can’t guarantee they will all stay unbroken.)
Anthropic (this is an example of llms-full.txt)
Cloudflare (llms-full.txt)
What’s Next?
On Marketing is a generative engine optimization agency built from the ground up to help marketers navigate the new world of AI SEO and LLM optimization. Contact us to learn how our team can help your B2B company excel in AI marketing.