PBQ: Master certifications with drag-and-drop, CLI & network simulations • 60+ Free Tools: PDF, Image, Network, Developer & Security utilities — all client-side • Practice Quizzes: CompTIA A+, Security+, Network+, Linux+ — exam-ready prep • OpenClaw: Your own 24/7 private AI assistant — managed server, live in 5 min • Tutorials: Step-by-step guides for Linux, cybersecurity, networking & system administration • PBQ: Master certifications with drag-and-drop, CLI & network simulations • 60+ Free Tools: PDF, Image, Network, Developer & Security utilities — all client-side • Practice Quizzes: CompTIA A+, Security+, Network+, Linux+ — exam-ready prep • OpenClaw: Your own 24/7 private AI assistant — managed server, live in 5 min • Tutorials: Step-by-step guides for Linux, cybersecurity, networking & system administration •
AiCybr logo AiCybr
🤖

Robots.txt Generator

Generate robots.txt files to control how search engines crawl your website. Include sitemap and custom rules.

No uploads • Client-side only • Your data stays on your device
Template
Sitemap URL (optional)

Generated robots.txt

# robots.txt

User-agent: *
Disallow:

📚 Common Examples

Block specific folder:
User-agent: *
Disallow: /admin/
Block specific bot:
User-agent: BadBot
Disallow: /
Block with exceptions:
User-agent: *
Disallow: /
Allow: /public/

📝 How to Use

  1. Select a template or write custom rules
  2. Add your sitemap URL (optional but recommended)
  3. Download or copy the generated file
  4. Upload robots.txt to your website root directory
  5. Test at https://yoursite.com/robots.txt

Related Tools

Meta Tags Generator

Your Privacy Guaranteed

All processing happens directly in your browser using JavaScript. Your files, passwords, and data never leave your device and are never uploaded to our servers. We cannot see, store, or access anything you process with this tool.

No account required • No tracking • No data collection • Works offline after initial load