# RobotsGate

> Free robots.txt generator, validator and live-site checker for AI crawlers such as GPTBot, ClaudeBot, Google-Extended and PerplexityBot, built on a sourced AI crawler registry. Public JSON API plus a remote MCP server for AI agents.

robots.txt is voluntary and is not access control: it only reaches crawlers that read and follow it. Crawler details come from each operator's own documentation (linked per crawler in the registry). RobotsGate is not affiliated with Meta or any other crawler or agent operator.

## For AI agents

- [MCP server](https://robotsgate.mike-tusa.workers.dev/mcp): remote MCP over Streamable HTTP at `https://robotsgate.mike-tusa.workers.dev/mcp` (POST only, stateless, JSON responses, no sessions). Protocol versions: 2026-07-28, 2025-11-25, 2025-06-18, 2025-03-26, 2024-11-05 (2026-07-28 per-request `_meta`; older versions use `initialize`). Tools: `list_crawlers` (optional `category`: training, dataset, search, user_fetch, other), `generate_robots` (`categories`, `agents`, `default`, `disallow_paths`, `allow_paths`, `sitemaps`, `custom_rules`, all optional), `validate_robots` (`robots_txt` required, up to 50,000 characters, and the whole request must fit in 65,536 bytes; optional `path`) and `check_site` (`url` required, optional `path`). All read-only. Client config (most MCP clients accept this; some use a different format, for example VS Code uses a `servers` key): `{"mcpServers":{"robotsgate":{"type":"http","url":"https://robotsgate.mike-tusa.workers.dev/mcp"}}}`
- [OpenAPI 3.1 spec](https://robotsgate.mike-tusa.workers.dev/openapi.json): machine-readable description of the JSON API
- [JSON API docs](https://robotsgate.mike-tusa.workers.dev/docs): human-readable API reference with examples
- [MCP server card](https://robotsgate.mike-tusa.workers.dev/.well-known/mcp/server-card.json): static description of the MCP server and its tools

## JSON API

- `GET https://robotsgate.mike-tusa.workers.dev/api/crawlers` (optional `?category=training|dataset|search|user_fetch|other`): the AI crawler registry (`crawlers[]` with `name`, `operator`, `category`, `respects_robots`, `docs_url`; plus `unverified`, `categories`, `notes`).
- `POST https://robotsgate.mike-tusa.workers.dev/api/generate` (JSON body: `categories`, `agents`, `default`, `disallow_paths`, `allow_paths`, `sitemaps`, `custom_rules`): `{ robots_txt, summary, notes, custom_rules_diagnostics, personal_agents_note }`.
- `POST https://robotsgate.mike-tusa.workers.dev/api/validate` (JSON body: `robots_txt` up to 512,000 bytes, optional `path`): `valid`, `errors`, `warnings`, `info`, `diagnostics`, `stats`, `ai_crawlers[]` (per crawler: `status` allowed/blocked, `deciding_rule`), `personal_agents`.
- `GET https://robotsgate.mike-tusa.workers.dev/api/check?url=example.com` (optional `&path=/`): fetches the site's /robots.txt (default ports only; private and internal addresses refused; cached 5 minutes) and returns the validate fields plus `http_status`, `final_url`, `robots_found`, `interpretation`.
- `GET https://robotsgate.mike-tusa.workers.dev/api/health`: `{ "ok": true, "version": "0.2.9" }`
- Errors: `{ "error": { "code", "message", "details"? } }` with HTTP 400 `invalid_input`/`invalid_json`/`url_not_allowed`, 405 `method_not_allowed`, 413 `payload_too_large`, 415 `unsupported_media_type`, 422 `too_complex`, 429 `rate_limited` (see `Retry-After`), 500 `internal_error`, 502 `dns_check_failed`/`upstream_error`/`too_many_redirects`, 504 `upstream_timeout`.

## Limits, keys and pricing

- Free (no key): about 20 `/api/check` requests and 60 `/api/generate` + `/api/validate` requests (shared) per 60 seconds per client IP (an IPv6 client counts per /64), counted per Cloudflare location. `/api/crawlers` has no per-client limit.
- RobotsGate Pro, $9/mo via Polar: 300 checks and 600 generate/validate requests per 60 seconds per license. Send `Authorization: Bearer RBTG-…` or `X-License-Key: RBTG-…`. A missing, invalid or expired key is not an error: the request just gets the free limits. [Subscribe](https://buy.polar.sh/polar_cl_pJjxZL8dDfTebUMNAEOnWNscJgTjNZPWbiIln3H3k6l)
- The MCP endpoint uses the same keys and the same counters: each `check_site` call counts as one `/api/check` request, and each `generate_robots` or `validate_robots` call counts against the shared generate/validate limit. `list_crawlers`, `initialize`, `tools/list` and other non-check messages don't use those counters, but have a light cap of about 120 per minute per IP (over it: HTTP 429 with `Retry-After`; for `list_crawlers`, a tool result with `isError: true`). Over an API limit, the tool result has `isError: true` with the reason and `retryAfterSeconds`.
- MCP request caps: 65,536-byte body, one `tools/call` per HTTP request, JSON-RPC batches (legacy versions only) of at most 10 messages.

## Guides

- [How to block GPTBot and other AI crawlers](https://robotsgate.mike-tusa.workers.dev/guides/block-gptbot-ai-crawlers)
- [Allow AI search, block AI training](https://robotsgate.mike-tusa.workers.dev/guides/allow-ai-search-block-ai-training)
- [Check which AI bots can crawl your site](https://robotsgate.mike-tusa.workers.dev/guides/check-which-ai-bots-can-crawl-your-site)
- [Cloudflare AI crawler settings and robots.txt](https://robotsgate.mike-tusa.workers.dev/guides/cloudflare-ai-crawler-settings-robots-txt)
- [Should I let Meta's Muse agents onto my site?](https://robotsgate.mike-tusa.workers.dev/guides/should-i-let-muse-agents-onto-my-site)

## Optional

- [Homepage, generator, validator and site check](https://robotsgate.mike-tusa.workers.dev/)
- Contact: digitalpromohub.support@gmail.com