Robots.txt Generator

This robots.txt generator lets you build a clean, valid robots.txt file visually — no need to memorize the syntax. Pick a user-agent, set allow or disallow paths, add your sitemaps, and copy or download the ready-to-use file. It is made for site owners, developers, and SEOs who want to control how search engines and AI crawlers access their site.

A robots.txt file sits at the root of your domain (for example https://example.com/robots.txt) and is one of the first things a crawler checks before indexing your pages. Start from a preset, paste an existing file to edit it, or build custom rule groups from scratch — the live preview updates as you go.

Private by design: the entire file is built in your browser. Nothing you type, import, or download is ever sent to a server.

How to Create a Robots.txt File

Start from a preset

Click a preset to load a common setup: Standard, Allow All, Block All, Block AI Bots, E-commerce, or Blank to build from scratch.

Choose a user-agent

For each rule group, pick a crawler from the dropdown (Googlebot, Bingbot, GPTBot, and more) or choose Custom to type any bot name. Use Add Rule Group to target several bots with different rules.

Set rules and crawl-delay

Add Allow or Disallow paths — type a path and pick from the suggestions for common directories. Optionally set a crawl-delay in seconds for each group.

Add your sitemaps

Use Add Sitemap URL to point crawlers at your sitemap files so they can discover your content faster. Add as many sitemap URLs as you need.

Copy or download

Review the live preview, then use Copy to grab the text or Download to save robots.txt. Upload it to your site's root directory.

Editing an existing file? Click Import Existing, paste your current robots.txt, and apply it. The tool parses it into editable rule groups so you can adjust the rules visually.

Features

Visual Rule Builder

Build rules through rule groups, each with a user-agent, allow/disallow paths, and an optional crawl-delay. Add or remove groups and rules with one click.

19 Built-in Bots

Choose from 19 preset user-agents covering search engines, social platforms, and AI crawlers, or enter any custom bot name.

Block AI Crawlers

One preset blocks GPTBot, ChatGPT-User, Google-Extended, CCBot, anthropic-ai, and Bytespider while still allowing regular search engines.

6 Preset Templates

Standard, Allow All, Block All, Block AI Bots, E-commerce, and Blank give you a working starting point in one click.

Allow & Disallow Paths

Set any path as Allow or Disallow, including wildcard patterns like /*.pdf$ that modern crawlers understand.

Path Suggestions

As you type, the tool suggests common directories like /admin/, /wp-admin/, /api/, and /cart/.

Crawl-delay Control

Set a crawl-delay in seconds per user-agent to ease server load from aggressive bots.

Multiple Sitemaps

Add one or more sitemap URLs so crawlers can find and index your content faster.

Import & Edit

Paste an existing robots.txt and the parser turns it into visual rule groups you can refine and re-export.

Live Syntax Preview

The output updates in real time with color-coded directives, values, and sitemap URLs for easy reading.

Copy & Download

Copy the file to your clipboard or download it as robots.txt, ready to upload to your site root.

Runs in Your Browser

All generating, importing, and exporting happens locally — your rules never leave your device.

Frequently Asked Questions

What is a robots.txt file?

A robots.txt file tells search engine crawlers which parts of your site they can or cannot access. It lives at the root of your domain and is one of the first files a crawler checks before indexing your pages.

How do I create a robots.txt file?

Start from a preset or a blank template, choose a user-agent, add your Allow or Disallow paths, add any sitemap URLs, then click Copy or Download. There is nothing to install — build the file visually and the preview shows the exact output.

Where do I put the robots.txt file?

Upload it to the root directory of your website so it is reachable at https://yourdomain.com/robots.txt. Crawlers look for that exact URL, so it must not sit in a subfolder.

How do I disallow all bots in robots.txt?

Use the Block All preset, which sets User-agent: * with Disallow: /. That tells every crawler not to crawl any page on your site, so use it with care.

Does robots.txt stop pages from being indexed?

Not exactly. Robots.txt blocks crawlers from accessing a page, but a blocked URL can still show in search results if other pages link to it. To fully keep a page out of search results, use a noindex meta tag instead.

How do I add a sitemap to robots.txt?

Click Add Sitemap URL and enter the full address of your sitemap (for example https://example.com/sitemap.xml). It is added as a Sitemap: line in the output. You can add several sitemap URLs.

How do I block AI bots from scraping my content?

Use the Block AI Bots preset. It allows regular search engines but adds Disallow rules for GPTBot, ChatGPT-User, Google-Extended, CCBot, anthropic-ai, and Bytespider.

What is Crawl-delay and can I use wildcards?

Crawl-delay tells a bot how many seconds to wait between requests, which can ease server load (note that Googlebot ignores it — use Search Console instead). For paths, most modern crawlers support * to match any sequence and $ to match the end of a URL, so /*.pdf$ blocks all PDF files.