We’ve added support for a custom llms.txt file—giving you more control over how your website content is accessed by AI language models. This file works similarly to robots.txt, but specifically for AI crawlers used by tools like ChatGPT, Perplexity, and others.
By defining rules in your llms.txt, you can choose to allow or block certain pages or directories from being indexed by AI models. This helps protect sensitive content, manage how your intellectual property is used, or simply decide what should and shouldn’t be part of AI training or responses.
You can easily manage this file in your project by going to Settings → General → SEO Settings, where you’ll find a field called “Contents of llms.txt”. Add your rules there, and it will automatically be served at yourdomain.com/llms.txt.
This update empowers you with better control over your website’s interaction with AI systems—simple, transparent, and fully under your command.