Control which AI crawlers may use your site, in robots.txt

RobotsGate is a free tool for building and checking robots.txt rules for AI crawlers. It uses a registry of crawler user-agent tokens, and each entry was checked against its operator's own documentation.

Please read first: robots.txt is a voluntary standard (RFC 9309). It asks crawlers to stay out. It cannot stop them, and RFC 9309 itself says it is "not a substitute for valid content security measures". Several operators say their user-triggered fetchers (for example ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) may not follow robots.txt. If you need to actually stop access, use authentication or server/CDN-level rules.

1. Generate robots.txt

Pick what each group of AI crawlers should do. "Leave out" adds no rule, so those crawlers follow your User-agent: * rules.

Per-crawler overrides

Loading registry…

Everyone else (User-agent: *) Paths to disallow for everyone (one per line, e.g. /admin/) Sitemap URL (optional) Custom rules (optional, appended as-is)

2. Validate robots.txt

Paste robots.txt Path to test

3. Check a live site

Fetches /robots.txt from the site you enter (public http/https hosts only) and shows which AI crawlers it allows for a path.

Site URL Path to test

RobotsGate Pro — $9/mo

Higher API rate limits for automation and scripting. Free tier stays available with no key.

Free: 20 /api/check requests per minute; 60 /api/generate + /api/validate per minute (per client IP).
Pro: 300 check / min and 600 generate+validate / min, keyed to your license.

After checkout, Polar emails a license key (prefix RBTG). Send it on API calls as:

Authorization: Bearer RBTG-…

Or use the X-License-Key header. An invalid or missing key is ignored and the free limits apply (no error).

Get RobotsGate Pro — $9/mo

For AI agents (MCP)

RobotsGate is also a remote Model Context Protocol (MCP) server with four read-only tools: list_crawlers, generate_robots, validate_robots and check_site. Most MCP clients accept this config; some use a different format (for example, VS Code uses a servers key). Clients with a settings screen just need the URL https://robotsgate.mike-tusa.workers.dev/mcp.

{ "mcpServers": { "robotsgate": { "type": "http", "url": "https://robotsgate.mike-tusa.workers.dev/mcp" } } }

Same free limits and Pro license key as the JSON API (details). Machine-readable docs: llms.txt · OpenAPI 3.1 spec

Guides

Where the crawler data comes from

Each crawler in the registry links to its operator's documentation and shows the date it was last checked. You can see the full list, including tokens we could not verify, at /api/crawlers. If you spot an error, please compare it against the operator's page linked there.