robots.txt Generator

Build robots.txt files with presets and a visual rule editor. Add user-agent blocks, disallow paths, and sitemap references. All processing happens in your browser.

Presets

Block PDF files (Disallow: /*.pdf$)

Sitemap URL

Crawl-delay (optional)

Preview

How it works

1. Pick a preset

Start with a preset for WordPress, Laravel, or a general setup. Then customize to fit your needs.

2. Add rules

Add user-agent blocks, disallow and allow paths, sitemap URL, and crawl delay.

3. Download

Copy the output or download as robots.txt. Upload to your site root directory.

Community Presets

No presets yet

Be the first to share a preset for this tool.

Share Your Preset

Related Tools

.htaccess Generator

Generate .htaccess rules for redirects, caching, and security.

Sitemap Validator

Validate XML sitemaps for protocol compliance and common issues.

Meta Tag Generator

Generate meta tags and Open Graph tags for better SEO.

Need more developer tools?

We're building more free utilities for web artisans. Get in touch with suggestions.