Free · runs in your browser

robots.txt Generator

Build robots.txt files with presets and a visual rule editor. Add user-agent blocks, disallow paths, and sitemap references. All processing happens in your browser.

no account no rate limit nothing stored

Input

Presets

Block PDF files (Disallow: /*.pdf$)

AI crawlers Block AI training GPTBot, ClaudeBot, Google-Extended and others that collect content for AI models. Your pages stay in regular search, including Google's AI Overviews. Block AI search and answers OAI-SearchBot, Claude-SearchBot, PerplexityBot and others. Your pages stop being cited in ChatGPT, Claude and Perplexity answers. Assistants that open a page for a user may ignore this.

Sitemap URL (full URL, including https://)

Crawl-delay (optional)

Output

// how it works

How it works

step 01

Pick a preset

Start with a preset for WordPress, Laravel, or a general setup. Then customize to fit your needs.

step 02

Add rules

Add user-agent blocks, disallow and allow paths, sitemap URL, and crawl delay.

step 03

Download

Copy the output or download as robots.txt. Upload to your site root directory.

// frequently asked

Common questions

Does robots.txt stop pages from being indexed?

Not reliably. Disallow only blocks crawling, not indexing. A disallowed URL can still appear in search results if it is linked from elsewhere. Use a noindex meta tag or header to keep a page out of the index.

Where does this file need to live?

At the root of your domain, for example example.com/robots.txt. It will not work from a subfolder.

Does blocking Google-Extended keep my site out of AI Overviews?

No. Google-Extended controls whether your content is used to train and ground Gemini, not Google Search. To keep pages out of AI Overviews and AI Mode, use the generative AI setting in Google Search Console instead.

Do I need this if I run Signocore SEO?

The plugin can manage the same file automatically. This tool is useful for previewing rules before you apply them, or for sites not on WordPress.

// community presets

Shared by users.

No presets yet

Be the first to share a preset for this tool.

Share your preset

Skip the copy-paste next time.

Signocore SEO generates this kind of schema and meta data automatically for every post. Free core, or €69 once for redirects, AI and advanced schema.

Get Signocore SEO Free tools